Oct 30 02:52:07.995683 ip-10-0-140-132 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 02:52:07.995695 ip-10-0-140-132 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 02:52:07.995702 ip-10-0-140-132 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 02:52:07.995974 ip-10-0-140-132 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 02:52:18.083069 ip-10-0-140-132 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 02:52:18.083084 ip-10-0-140-132 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3c259e25cea644619cfce73a99c9c63d --
Oct 30 02:54:12.126202 ip-10-0-140-132 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:54:12.587293 ip-10-0-140-132 kubenswrapper[2544]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:12.587293 ip-10-0-140-132 kubenswrapper[2544]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:54:12.587293 ip-10-0-140-132 kubenswrapper[2544]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:12.587293 ip-10-0-140-132 kubenswrapper[2544]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:12.591645 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591553    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:12.591645 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591643    2544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:12.591645 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591647    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:12.591645 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591650    2544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:12.591645 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591653    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591656    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591658    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591661    2544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591665    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591667    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591670    2544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591672    2544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591675    2544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591677    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591680    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591683    2544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591686    2544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591689    2544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591691    2544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591694    2544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591696    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591699    2544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591702    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:12.591870 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591704    2544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591707    2544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591709    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591711    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591714    2544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591717    2544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591719    2544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591722    2544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591726    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591728    2544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591731    2544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591733    2544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591736    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591738    2544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591741    2544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591744    2544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591746    2544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591749    2544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591751    2544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591754    2544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:12.592318 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591756    2544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591759    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591761    2544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591763    2544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591766    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591770    2544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591774    2544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591778    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591781    2544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591783    2544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591785    2544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591788    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591792    2544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591794    2544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591797    2544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591799    2544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591802    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591805    2544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591807    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:12.592789 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591811    2544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591813    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591816    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591818    2544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591821    2544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591824    2544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591826    2544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591828    2544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591831    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591833    2544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591836    2544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591838    2544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591840    2544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591843    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591845    2544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591848    2544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591850    2544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591853    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591855    2544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591875    2544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:12.593282 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591880    2544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591885    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591890    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591893    2544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591896    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591899    2544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591901    2544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591904    2544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591907    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591911    2544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591915    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591918    2544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591922    2544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591925    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591928    2544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591931    2544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591933    2544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591936    2544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591939    2544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:12.593745 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591941    2544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591945    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591948    2544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591950    2544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.591953    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592732    2544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592738    2544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592741    2544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592744    2544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592747    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592749    2544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592753    2544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592757    2544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592760    2544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592763    2544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592766    2544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592769    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592772    2544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592775    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592777    2544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:12.594255 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592780    2544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592783    2544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592786    2544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592788    2544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592791    2544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592794    2544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592797    2544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592799    2544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592802    2544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592805    2544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592807    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592810    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592812    2544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592815    2544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592817    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592819    2544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592822    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592824    2544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592827    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592829    2544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:12.594732 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592832    2544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592834    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592837    2544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592839    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592842    2544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592845    2544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592847    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592849    2544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592853    2544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592855    2544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592876    2544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592880    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592884    2544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592887    2544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592891    2544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592893    2544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592896    2544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592898    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592901    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592904    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:12.595230 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592906    2544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592909    2544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592912    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592915    2544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592918    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592920    2544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592922    2544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592925    2544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592927    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592930    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592932    2544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592935    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592937    2544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592940    2544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592943    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592945    2544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592948    2544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592951    2544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592953    2544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592956    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:12.595696 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592958    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592962    2544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592967    2544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592970    2544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592972    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592975    2544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592978    2544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592980    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592982    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592987    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592990    2544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592992    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592995    2544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.592997    2544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593000    2544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593002    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593005    2544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593008    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593010    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:12.596176 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593013    2544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593015    2544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593018    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593020    2544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593022    2544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593025    2544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593027    2544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593030    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593033    2544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593035    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593037    2544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593041    2544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593578    2544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593584    2544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593588    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593593    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593596    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593599    2544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593602    2544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:12.596625 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593604    2544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593607    2544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593609    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593613    2544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593616    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593618    2544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593621    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593623    2544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593626    2544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593628    2544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593631    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593633    2544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593636    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593638    2544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593641    2544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593643    2544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593646    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593649    2544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593651    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593653    2544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:12.597125 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593656    2544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593659    2544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593661    2544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593664    2544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593666    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593669    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593671    2544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593674    2544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593678    2544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593681    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593683    2544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593686    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593688    2544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593691    2544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593693    2544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593697    2544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593700    2544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593702    2544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593704    2544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:12.597594 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593707    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593709    2544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593712    2544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593714    2544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593716    2544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593719    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593721    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593724    2544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593726    2544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593729    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593731    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593734    2544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593736    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593738    2544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593741    2544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593744    2544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593749    2544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593752    2544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593754    2544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593757    2544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:12.598052 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593760    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593763    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593766    2544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593769    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593772    2544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593776    2544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593779    2544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593782    2544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593786    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593788    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593791    2544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593794    2544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593797    2544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593800    2544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593802    2544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593804    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593807    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593809    2544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593812    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:12.598524 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593814    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593817    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593820    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593823    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593825    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593827    2544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593830    2544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593832    2544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593834    2544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593837    2544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593839    2544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593842    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593844    2544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593846    2544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593849    2544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593852    2544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593855    2544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593872    2544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593874    2544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593877    2544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:12.598969 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.593880    2544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.593977    2544 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.593984    2544 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.593996    2544 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594003    2544 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594007    2544 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594010    2544 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594014    2544 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594019    2544 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594022    2544 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594025    2544 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594028    2544 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594032    2544 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594035    2544 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594038    2544 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594040    2544 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594043    2544 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594046    2544 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594049    2544 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594054    2544 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594057    2544 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594060    2544 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594063    2544 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594066    2544 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:54:12.599439 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594070    2544 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594073    2544 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594076    2544 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594080    2544 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594083    2544 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594086    2544 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594089    2544 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594092    2544 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594095    2544 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594103    2544 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594106    2544 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594109    2544 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594112    2544 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594115    2544 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594118    2544 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594122    2544 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594125    2544 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594128    2544 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594131    2544 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594133    2544 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594137    2544 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594140    2544 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594143    2544 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594146    2544 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594149    2544 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:54:12.600052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594152    2544 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594155    2544 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594157    2544 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594160    2544 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594163    2544 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594166    2544 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594170    2544 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594173    2544 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594176    2544 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594179    2544 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594182    2544 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594185    2544 flags.go:64] FLAG: --help="false"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594188    2544 flags.go:64] FLAG: --hostname-override="ip-10-0-140-132.ec2.internal"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594192    2544 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594195    2544 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594198    2544 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594201    2544 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594204    2544 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594207    2544 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594209    2544 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594212    2544 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594215    2544 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594218    2544 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594221    2544 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:54:12.600700 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594223    2544 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594226    2544 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594230    2544 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594233    2544 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594236    2544 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594239    2544 flags.go:64] FLAG: --lock-file=""
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594242    2544 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594245    2544 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594248    2544 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594253    2544 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594256    2544 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594259    2544 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594262    2544 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594265    2544 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594269    2544 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594273    2544 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594276    2544 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594280    2544 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594284    2544 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594288    2544 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594291    2544 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594294    2544 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594298    2544 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594300    2544 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594303    2544 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:54:12.601287 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594306    2544 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594309    2544 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594316    2544 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594319    2544 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594323    2544 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594326    2544 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594328    2544 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594331    2544 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594334    2544 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594338    2544 flags.go:64] FLAG: --port="10250"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594341    2544 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594344    2544 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06a63c2ebcf15d44f"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594347    2544 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594350    2544 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594353    2544 flags.go:64] FLAG: --register-node="true"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594356    2544 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594359    2544 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594362    2544 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594365    2544 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594368    2544 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594372    2544 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594375    2544 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594379    2544 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594382    2544 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594385    2544 flags.go:64] FLAG: --runonce="false"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594387    2544 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:54:12.601871 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594390    2544 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594393    2544 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594396    2544 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594399    2544 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594403    2544 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594406    2544 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594409    2544 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594412    2544 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594414    2544 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594417    2544 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594420    2544 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594423    2544 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594426    2544 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594429    2544 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594434    2544 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594437    2544 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594440    2544 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594445    2544 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594448    2544 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594450    2544 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594453    2544 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594456    2544 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594459    2544 flags.go:64] FLAG: --v="2"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594463    2544 flags.go:64] FLAG: --version="false"
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594467    2544 flags.go:64] FLAG: --vmodule=""
Oct 30 02:54:12.602493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594471    2544 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594474    2544 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594603    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594607    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594610    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594613    2544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594615    2544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594618    2544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594621    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594624    2544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594626    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594628    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594631    2544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594634    2544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594636    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594639    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594641    2544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594644    2544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594646    2544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:12.603099 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594648    2544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594651    2544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594653    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594657    2544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594659    2544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594662    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594664    2544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594666    2544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594669    2544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594671    2544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594674    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594676    2544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594678    2544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594681    2544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594683    2544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594686    2544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594689    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594692    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594701    2544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594703    2544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:12.603545 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594706    2544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594709    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594711    2544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594714    2544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594716    2544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594719    2544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594722    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594724    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594727    2544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594729    2544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594732    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594734    2544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594737    2544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594739    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594742    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594746    2544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594748    2544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594751    2544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594753    2544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594755    2544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:12.604025 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594758    2544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594762    2544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594766    2544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594769    2544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594771    2544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594774    2544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594776    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594779    2544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594782    2544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594784    2544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594787    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594790    2544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594792    2544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594794    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594797    2544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594799    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594802    2544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594804    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594807    2544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594810    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:12.604504 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594812    2544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594814    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594817    2544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594819    2544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594822    2544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594824    2544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594826    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594831    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594833    2544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594836    2544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594838    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594841    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594844    2544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594846    2544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594849    2544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594851    2544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594854    2544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594869    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594872    2544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:12.604993 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594875    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594879    2544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594881    2544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594885    2544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594888    2544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594891    2544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594894    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594896    2544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594899    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594902    2544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.594927    2544 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594969    2544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594973    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594977    2544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594980    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:12.605428 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594983    2544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594985    2544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594988    2544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.594990    2544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595000    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595004    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595008    2544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595011    2544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595014    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595017    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595019    2544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595022    2544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595024    2544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595027    2544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595030    2544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595033    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595035    2544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595038    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595042    2544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:12.605818 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595044    2544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595048    2544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595052    2544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595055    2544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595058    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595061    2544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595064    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595067    2544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595070    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595073    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595076    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595078    2544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595081    2544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595084    2544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595087    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595090    2544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595092    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595095    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595097    2544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:12.606285 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595101    2544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595103    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595106    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595108    2544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595111    2544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595113    2544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595115    2544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595118    2544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595120    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595123    2544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595125    2544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595128    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595131    2544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595134    2544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595136    2544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595139    2544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595141    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595144    2544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595146    2544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:12.606738 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595149    2544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595151    2544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595154    2544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595156    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595158    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595161    2544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595163    2544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595166    2544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595169    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595171    2544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595173    2544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595176    2544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595178    2544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595182    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595185    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595187    2544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595190    2544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595192    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595195    2544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595197    2544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:12.607205 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595200    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595202    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595205    2544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595207    2544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595210    2544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595214    2544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595216    2544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595219    2544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595223    2544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595226    2544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595229    2544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595232    2544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595235    2544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595238    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595241    2544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595244    2544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595247    2544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595250    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595252    2544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595255    2544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:12.607670 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595257    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595260    2544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595263    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595266    2544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.595268    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.603540    2544 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.603555    2544 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.605021    2544 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.605029    2544 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605101    2544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605107    2544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605118    2544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605121    2544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605124    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605127    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605130    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605133    2544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605135    2544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605138    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:12.608141 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605141    2544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605143    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605146    2544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605149    2544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605151    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605154    2544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605156    2544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605159    2544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605161    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605164    2544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605166    2544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605171    2544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605174    2544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605184    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605187    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605190    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605192    2544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605196    2544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605198    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605201    2544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:12.608591 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605204    2544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605206    2544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605209    2544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605211    2544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605214    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605217    2544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605219    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605222    2544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605225    2544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605227    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605229    2544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605232    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605234    2544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605237    2544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605239    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605242    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605244    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605248    2544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605252    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:12.609061 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605255    2544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605257    2544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605260    2544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605262    2544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605265    2544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605267    2544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605270    2544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605279    2544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605281    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605284    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605286    2544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605290    2544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605292    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605295    2544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605297    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605300    2544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605302    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605305    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605307    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605309    2544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:12.609494 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605312    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605314    2544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605316    2544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605319    2544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605321    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605324    2544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605326    2544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605329    2544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605332    2544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605334    2544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605337    2544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605339    2544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605342    2544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605344    2544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605347    2544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605349    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605351    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605354    2544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605356    2544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605359    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:12.609972 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605367    2544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605369    2544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605372    2544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605374    2544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605377    2544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605379    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605381    2544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605384    2544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605386    2544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605388    2544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605391    2544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605393    2544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605395    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605398    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605400    2544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605403    2544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:12.610441 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605405    2544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.605432    2544 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605645    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605650    2544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605653    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605656    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605659    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605662    2544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605665    2544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605668    2544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605671    2544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605673    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605676    2544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605678    2544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605681    2544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605683    2544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:12.610836 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605685    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605695    2544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605697    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605700    2544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605705    2544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605709    2544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605712    2544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605714    2544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605717    2544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605719    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605722    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605724    2544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605727    2544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605730    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605732    2544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605735    2544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605737    2544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605740    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605742    2544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:12.611229 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605745    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605747    2544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605750    2544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605752    2544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605755    2544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605757    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605760    2544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605762    2544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605764    2544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605767    2544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605770    2544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605772    2544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605775    2544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605777    2544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605780    2544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605791    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605794    2544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605796    2544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605798    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605801    2544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:12.611658 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605803    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605806    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605808    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605810    2544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605813    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605815    2544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605818    2544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605820    2544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605823    2544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605826    2544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605828    2544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605831    2544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605834    2544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605837    2544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605839    2544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605842    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605844    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605846    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605849    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:12.612133 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605851    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605854    2544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605876    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605880    2544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605882    2544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605885    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605888    2544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605890    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605893    2544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605902    2544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605905    2544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605908    2544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605910    2544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605912    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605915    2544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605917    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605919    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605922    2544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605925    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605927    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:12.612575 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605929    2544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605932    2544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605934    2544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605937    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605940    2544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605944    2544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605947    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605949    2544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605952    2544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605955    2544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605957    2544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605959    2544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605962    2544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.605965    2544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.605988    2544 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606045    2544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:12.613053 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606048    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606051    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606054    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606057    2544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606059    2544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606062    2544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606071    2544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606073    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606076    2544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606078    2544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606081    2544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606083    2544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606092    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606095    2544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606097    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606100    2544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606104    2544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606108    2544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606110    2544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:12.613424 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606113    2544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606115    2544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606118    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606120    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606123    2544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606125    2544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606128    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606131    2544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606133    2544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606135    2544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606138    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606140    2544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606142    2544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606145    2544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606147    2544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606150    2544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606152    2544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606155    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606157    2544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606160    2544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:12.613979 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606168    2544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606171    2544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606174    2544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606176    2544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606179    2544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606181    2544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606183    2544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606186    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606188    2544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606191    2544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606193    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606196    2544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606198    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606200    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606203    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606205    2544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606207    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606210    2544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606212    2544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:12.614454 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606215    2544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606217    2544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606219    2544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606222    2544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606224    2544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606227    2544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606229    2544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606231    2544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606234    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606236    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606239    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606241    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606244    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606246    2544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606254    2544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606257    2544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606259    2544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606262    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606264    2544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:12.614905 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606266    2544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606269    2544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606271    2544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606274    2544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606276    2544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606278    2544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606281    2544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606283    2544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606286    2544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606289    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606291    2544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606294    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606297    2544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606299    2544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606302    2544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606305    2544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606307    2544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606311    2544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606314    2544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:12.615349 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606317    2544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606320    2544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606326    2544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606329    2544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606331    2544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606334    2544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606336    2544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606338    2544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:12.606341    2544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.606397    2544 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.608891    2544 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.609839    2544 server.go:1014] "Starting client certificate rotation"
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.609947    2544 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.610707    2544 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:54:12.615782 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.612212    2544 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:54:12.616619 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.616605    2544 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:54:12.633608 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.633585    2544 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:54:12.637479 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.637462    2544 log.go:25] "Validated CRI v1 image API"
Oct 30 02:54:12.638332 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.638308    2544 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:54:12.642409 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.642379    2544 fs.go:135] Filesystem UUIDs: map[399f0473-eba9-4178-b219-e4f2de6981bc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ba937578-9349-4f84-bb84-fdccb72db4e5:/dev/nvme0n1p4]
Oct 30 02:54:12.642472 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.642407    2544 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 02:54:12.647781 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.647657    2544 manager.go:217] Machine: {Timestamp:2026-10-30 02:54:12.646186893 +0000 UTC m=+0.386668325 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3128802 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a727b0c0c79c9f293d059253588ee SystemUUID:ec2a727b-0c0c-79c9-f293-d059253588ee BootID:3c259e25-cea6-4461-9cfc-e73a99c9c63d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e4:70:05:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e4:70:05:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:00:c8:3f:0b:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:54:12.647781 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.647777    2544 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:54:12.647899 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.647884    2544 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:54:12.651300 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.651264    2544 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:54:12.651445 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.651303    2544 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:54:12.651496 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.651448    2544 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:54:12.651496 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.651457    2544 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:54:12.651496 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.651469    2544 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:54:12.652295 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.652283    2544 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:54:12.653497 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.653486    2544 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:54:12.657949 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.657935    2544 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:54:12.658079 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.658070    2544 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:54:12.658898 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.658888    2544 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:54:12.658929 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.658903    2544 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:54:12.658929 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.658913    2544 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:54:12.658929 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.658922    2544 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:54:12.659030 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.658930    2544 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:54:12.659371 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.659358    2544 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:54:12.662145 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.662119    2544 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:54:12.668869 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.668840    2544 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:54:12.669789 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.669777    2544 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:54:12.669827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.669807    2544 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:54:12.669993 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.669981    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:54:12.670031 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.669998    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:54:12.670031 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.670006    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:54:12.670031 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.670012    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:54:12.670031 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.670018    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:54:12.670031 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.670023    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:54:12.670756 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.670747    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:54:12.670791 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.670756    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:54:12.670791 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.670763    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:54:12.670791 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.670770    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:54:12.670791 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.670778    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:54:12.670791 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.670786    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:54:12.671829 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.671819    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:54:12.671829 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.671829    2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:54:12.675813 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.675797    2544 server.go:1263] "Started kubelet"
Oct 30 02:54:12.676202 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.675989    2544 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:54:12.676202 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.676113    2544 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:54:12.676202 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.676169    2544 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:54:12.676637 ip-10-0-140-132 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:54:12.677729 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.677709    2544 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:54:12.679951 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.679925    2544 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:54:12.682707 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.682686    2544 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 02:54:12.685525 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.685509    2544 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:54:12.685598 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.685526    2544 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:54:12.685650 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.685591    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:12.686285 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.686264    2544 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:54:12.686285 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.686288    2544 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:54:12.686440 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.686364    2544 factory.go:55] Registering systemd factory
Oct 30 02:54:12.686440 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.686380    2544 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:54:12.686440 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.686389    2544 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:54:12.686440 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.686399    2544 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:54:12.686440 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.686437    2544 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:54:12.686635 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.686615    2544 factory.go:153] Registering CRI-O factory
Oct 30 02:54:12.686635 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.686632    2544 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:54:12.686719 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.686696    2544 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:54:12.686719 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.686717    2544 factory.go:103] Registering Raw factory
Oct 30 02:54:12.686811 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.686734    2544 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:54:12.688002 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.687987    2544 manager.go:319] Starting recovery of all containers
Oct 30 02:54:12.698033 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.698016    2544 manager.go:324] Recovery completed
Oct 30 02:54:12.702597 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.702584    2544 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:12.705045 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.705024    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:12.705143 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.705062    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:12.705143 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.705076    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:12.705605 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.705593    2544 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:54:12.705651 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.705605    2544 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:54:12.705651 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.705622    2544 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:54:12.705651 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.705640    2544 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:12.706301 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.706288    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:12.706351 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.706314    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:12.706351 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.706326    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:12.707802 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.707791    2544 policy_none.go:50] "Start"
Oct 30 02:54:12.707839 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.707806    2544 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:54:12.707839 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.707816    2544 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:54:12.708992 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.708974    2544 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:54:12.709145 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.709117    2544 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 02:54:12.709145 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.709119    2544 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-132.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 02:54:12.709790 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.709779    2544 policy_none.go:44] "Start"
Oct 30 02:54:12.720891 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.719706    2544 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-132.ec2.internal.18e32f170d328e92  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-132.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-132.ec2.internal,},FirstTimestamp:2026-10-30 02:54:12.675759762 +0000 UTC m=+0.416241190,LastTimestamp:2026-10-30 02:54:12.675759762 +0000 UTC m=+0.416241190,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-132.ec2.internal,}"
Oct 30 02:54:12.727110 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.727089    2544 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 02:54:12.727373 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.727352    2544 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 02:54:12.736082 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.736062    2544 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mwq4h"
Oct 30 02:54:12.744773 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.744754    2544 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mwq4h"
Oct 30 02:54:12.748253 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.748222    2544 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:54:12.750754 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.750741    2544 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:54:12.750816 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.750777    2544 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:54:12.750816 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.750789    2544 server.go:87] "Starting device plugin registration server"
Oct 30 02:54:12.751063 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.751051    2544 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:54:12.751100 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.751065    2544 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:54:12.751499 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.751171    2544 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:54:12.751499 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.751457    2544 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:54:12.751499 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.751467    2544 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:54:12.751792 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.751773    2544 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:54:12.751841 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.751818    2544 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:12.759700 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.759680    2544 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 02:54:12.759784 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.759708    2544 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-132.ec2.internal.18e32f170ef168f9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-132.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-132.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-132.ec2.internal,},FirstTimestamp:2026-10-30 02:54:12.705044729 +0000 UTC m=+0.445526159,LastTimestamp:2026-10-30 02:54:12.705044729 +0000 UTC m=+0.445526159,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-132.ec2.internal,}"
Oct 30 02:54:12.842260 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.842184    2544 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:54:12.844388 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.843470    2544 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:54:12.844388 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.843489    2544 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:54:12.844388 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.843510    2544 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:54:12.844388 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.843542    2544 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 02:54:12.851675 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.851652    2544 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:12.852614 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.852585    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:12.852727 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.852628    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:12.852727 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.852639    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:12.852727 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.852662    2544 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-132.ec2.internal"
Oct 30 02:54:12.858511 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.858497    2544 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:12.874165 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.874136    2544 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-132.ec2.internal"
Oct 30 02:54:12.874165 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.874168    2544 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-132.ec2.internal\": node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:12.918757 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.918727    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:12.943633 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.943592    2544 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"]
Oct 30 02:54:12.943803 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.943674    2544 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:12.944665 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.944648    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:12.944750 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.944677    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:12.944750 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.944689    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:12.946954 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.946941    2544 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:12.947130 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.947114    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:12.947174 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.947149    2544 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:12.947731 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.947716    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:12.947788 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.947748    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:12.947788 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.947758    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:12.947788 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.947720    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:12.947905 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.947802    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:12.947905 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.947811    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:12.949965 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.949947    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:12.950052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.949979    2544 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:12.950657 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.950644    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:12.950719 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.950666    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:12.950719 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.950680    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:12.968339 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.968317    2544 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-132.ec2.internal\" not found" node="ip-10-0-140-132.ec2.internal"
Oct 30 02:54:12.970623 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:12.970606    2544 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-132.ec2.internal\" not found" node="ip-10-0-140-132.ec2.internal"
Oct 30 02:54:12.988619 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.988588    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6b355a5b292f9d5c5019135ffc6b2ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"d6b355a5b292f9d5c5019135ffc6b2ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:12.988619 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.988620    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7470c725c1ecc41c5e63c668787a572e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-132.ec2.internal\" (UID: \"7470c725c1ecc41c5e63c668787a572e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:12.988744 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:12.988637    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6b355a5b292f9d5c5019135ffc6b2ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"d6b355a5b292f9d5c5019135ffc6b2ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:13.019000 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:13.018972    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:13.089769 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.089737    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7470c725c1ecc41c5e63c668787a572e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-132.ec2.internal\" (UID: \"7470c725c1ecc41c5e63c668787a572e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:13.089841 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.089773    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6b355a5b292f9d5c5019135ffc6b2ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"d6b355a5b292f9d5c5019135ffc6b2ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:13.089841 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.089790    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6b355a5b292f9d5c5019135ffc6b2ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"d6b355a5b292f9d5c5019135ffc6b2ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:13.089938 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.089877    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7470c725c1ecc41c5e63c668787a572e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-132.ec2.internal\" (UID: \"7470c725c1ecc41c5e63c668787a572e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:13.089938 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.089909    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6b355a5b292f9d5c5019135ffc6b2ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"d6b355a5b292f9d5c5019135ffc6b2ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:13.089938 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.089886    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6b355a5b292f9d5c5019135ffc6b2ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"d6b355a5b292f9d5c5019135ffc6b2ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:13.119121 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:13.119042    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:13.219396 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:13.219358    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:13.271750 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.271717    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:13.273836 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.273820    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:13.319845 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:13.319807    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:13.420445 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:13.420377    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:13.520892 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:13.520837    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:13.610476 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.610443    2544 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 02:54:13.610973 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.610624    2544 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 02:54:13.621786 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:13.621752    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:13.722280 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:13.722218    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:13.750689 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.750652    2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:49:14 +0000 UTC" deadline="2028-07-18 17:43:12.308227378 +0000 UTC"
Oct 30 02:54:13.750689 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.750682    2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15062h48m58.557548773s"
Oct 30 02:54:13.822955 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:13.822920    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:13.879086 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:13.879054    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b355a5b292f9d5c5019135ffc6b2ed.slice/crio-23ecde7fac698ec0024bd1926e08b12e03559b74c5193216dd96d14b6c2cd995 WatchSource:0}: Error finding container 23ecde7fac698ec0024bd1926e08b12e03559b74c5193216dd96d14b6c2cd995: Status 404 returned error can't find the container with id 23ecde7fac698ec0024bd1926e08b12e03559b74c5193216dd96d14b6c2cd995
Oct 30 02:54:13.884491 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.884469    2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:54:13.886116 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:13.886092    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7470c725c1ecc41c5e63c668787a572e.slice/crio-1e04b37ed35dee6a9eb88ba0131903679d47401b4cd7a7dc4e5a921f400897aa WatchSource:0}: Error finding container 1e04b37ed35dee6a9eb88ba0131903679d47401b4cd7a7dc4e5a921f400897aa: Status 404 returned error can't find the container with id 1e04b37ed35dee6a9eb88ba0131903679d47401b4cd7a7dc4e5a921f400897aa
Oct 30 02:54:13.899466 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.899446    2544 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:13.901450 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.901429    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:13.901527 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.901469    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:13.901527 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.901480    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:13.923053 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:13.923024    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:13.945712 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:13.945691    2544 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:14.023749 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:14.023691    2544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:14.066450 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:14.066421    2544 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:14.086498 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:14.086473    2544 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:14.114061 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:14.114041    2544 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:54:14.115752 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:14.115738    2544 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:14.136406 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:14.136384    2544 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:54:14.235335 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:14.235308    2544 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:16.221219 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.221190    2544 apiserver.go:52] "Watching apiserver"
Oct 30 02:54:16.247113 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.247089    2544 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 02:54:16.258978 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.258946    2544 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:54:16.259407 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.259378    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t","openshift-image-registry/node-ca-stg5z","openshift-multus/multus-fbpcs","openshift-dns/node-resolver-k6c24","openshift-network-operator/iptables-alerter-6th7l","openshift-ovn-kubernetes/ovnkube-node-244kd","kube-system/konnectivity-agent-x5wjw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal","openshift-multus/multus-additional-cni-plugins-hlhb7","openshift-multus/network-metrics-daemon-k6rks","kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal","openshift-cluster-node-tuning-operator/tuned-l6rrk","openshift-network-diagnostics/network-check-target-xg9mx","kube-system/global-pull-secret-syncer-ffnvh"]
Oct 30 02:54:16.261973 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.261951    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:16.264213 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.264194    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:16.264343 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.264320    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:16.266241 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.266221    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.268409 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.268389    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:16.271458 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.271440    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:16.274053 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.274036    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.276844 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.276825    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.279164 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.279145    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.280667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.280648    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:54:16.280866 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.280852    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:54:16.280939 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.280927    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f9m7f\""
Oct 30 02:54:16.281869 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.281442    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:16.284438 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.284420    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:16.284532 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.284497    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:16.287413 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.286699    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:16.287413 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.286761    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:16.289582 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.289564    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.305315 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.305296    2544 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:54:16.305406 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.305352    2544 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-svxpb"
Oct 30 02:54:16.317343 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.317320    2544 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-svxpb"
Oct 30 02:54:16.348533 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.348510    2544 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:54:16.358134 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358072    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.358134 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358112    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-device-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.358312 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358176    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9r6z\" (UniqueName: \"kubernetes.io/projected/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-kube-api-access-p9r6z\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.358312 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358222    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2811f6f-7a04-455e-953c-18c1018672d8-konnectivity-ca\") pod \"konnectivity-agent-x5wjw\" (UID: \"e2811f6f-7a04-455e-953c-18c1018672d8\") " pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:16.358312 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358272    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb190b5e-126c-4d02-9c3a-35dc5c10d521-iptables-alerter-script\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:16.358428 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358310    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb190b5e-126c-4d02-9c3a-35dc5c10d521-host-slash\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:16.358428 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358347    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-var-lib-kubelet\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.358428 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358372    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-k8s-cni-cncf-io\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.358428 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358396    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-cni-multus\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.358428 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358419    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-conf-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.358578 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358486    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-node-log\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.358578 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358538    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2811f6f-7a04-455e-953c-18c1018672d8-agent-certs\") pod \"konnectivity-agent-x5wjw\" (UID: \"e2811f6f-7a04-455e-953c-18c1018672d8\") " pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:16.358698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358579    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-kubernetes\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.358698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358596    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f8769fd-d1f7-4e86-b367-e5cc0e707174-cni-binary-copy\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.358698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358641    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhzw7\" (UniqueName: \"kubernetes.io/projected/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-kube-api-access-nhzw7\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:16.358698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358659    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-log-socket\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.358698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358673    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovnkube-config\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.358698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358688    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgdqh\" (UniqueName: \"kubernetes.io/projected/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-kube-api-access-vgdqh\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.358982 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358735    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5r4v\" (UniqueName: \"kubernetes.io/projected/95a07519-99bf-47ac-b236-c642491d5648-kube-api-access-s5r4v\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:16.358982 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358757    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysctl-d\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.358982 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358782    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-systemd\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.358982 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358825    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-cni-bin\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.358982 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358860    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-sys-fs\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.358982 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358894    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-kubelet-config\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:16.358982 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358928    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77q9p\" (UniqueName: \"kubernetes.io/projected/49a42451-a2db-456c-93ed-e0b932f13751-kube-api-access-77q9p\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.358982 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.358970    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvgls\" (UniqueName: \"kubernetes.io/projected/971e53b3-ff90-4be0-bf64-9c1013099b9d-kube-api-access-hvgls\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359002    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359026    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359054    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-host\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359093    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysctl-conf\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359110    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-kubelet\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359136    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-env-overrides\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359173    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cnibin\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359203    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-hosts-file\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359218    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359236    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-cni-bin\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359257    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359286    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-system-cni-dir\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359303    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m9gn\" (UniqueName: \"kubernetes.io/projected/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-kube-api-access-9m9gn\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359321    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49a42451-a2db-456c-93ed-e0b932f13751-etc-tuned\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.359368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359337    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-multus-certs\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359380    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58r88\" (UniqueName: \"kubernetes.io/projected/eb190b5e-126c-4d02-9c3a-35dc5c10d521-kube-api-access-58r88\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359411    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359450    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-hostroot\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359477    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-slash\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359506    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-etc-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359531    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-serviceca\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359554    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-host\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359579    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2f9l\" (UniqueName: \"kubernetes.io/projected/5f8769fd-d1f7-4e86-b367-e5cc0e707174-kube-api-access-j2f9l\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359632    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-kubelet\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359666    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-run-ovn-kubernetes\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359699    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovn-node-metrics-cert\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359727    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovnkube-script-lib\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359781    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359814    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-modprobe-d\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359847    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysconfig\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.360075 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359885    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-lib-modules\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359926    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-sys\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.359960    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-systemd-units\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360007    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-systemd\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360042    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-ovn\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360069    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-socket-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360108    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a42451-a2db-456c-93ed-e0b932f13751-tmp\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360145    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-cni-netd\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360182    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-etc-kubernetes\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360214    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-run-netns\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360239    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-registration-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360279    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-dbus\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360352    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-system-cni-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360380    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-cni-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360430    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-cnibin\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360467    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-var-lib-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360494    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.360815 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360537    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-run\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.361581 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360565    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55b4z\" (UniqueName: \"kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z\") pod \"network-check-target-xg9mx\" (UID: \"23d4d339-696a-4b94-b34f-19852eb3cbc2\") " pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:16.361581 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360623    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-socket-dir-parent\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.361581 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360656    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-daemon-config\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.361581 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360686    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-tmp-dir\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:16.361581 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360721    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.361581 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360755    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.361581 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.360784    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-netns\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.381598 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.381560    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:54:16.381801 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.381650    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:54:16.381801 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.381668    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:54:16.381801 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.381680    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:54:16.381801 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.381705    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:54:16.381801 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.381768    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:16.381801 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.381793    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vnlvm\""
Oct 30 02:54:16.382135 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.381810    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:16.382135 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382022    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:54:16.382135 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382052    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:54:16.382297 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382134    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:54:16.382297 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382149    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:54:16.382297 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382171    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:54:16.382297 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382188    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:16.382297 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382200    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:54:16.382297 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382232    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382315    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382371    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382380    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382423    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vxw8k\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382437    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382441    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382462    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z4s4b\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382425    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382483    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382508    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rg86w\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382423    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-sdsc6\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382544    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382555    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382601    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tm8xk\""
Oct 30 02:54:16.382667 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.382675    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5k8rj\""
Oct 30 02:54:16.389520 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.389503    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:54:16.389664 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.389560    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cr2zp\""
Oct 30 02:54:16.409664 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.409598    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal" event={"ID":"7470c725c1ecc41c5e63c668787a572e","Type":"ContainerStarted","Data":"1e04b37ed35dee6a9eb88ba0131903679d47401b4cd7a7dc4e5a921f400897aa"}
Oct 30 02:54:16.410918 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.410894    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal" event={"ID":"d6b355a5b292f9d5c5019135ffc6b2ed","Type":"ContainerStarted","Data":"23ecde7fac698ec0024bd1926e08b12e03559b74c5193216dd96d14b6c2cd995"}
Oct 30 02:54:16.431084 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.431061    2544 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:16.462005 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.461978    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovnkube-config\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.462159 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462020    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdqh\" (UniqueName: \"kubernetes.io/projected/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-kube-api-access-vgdqh\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.462159 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462048    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5r4v\" (UniqueName: \"kubernetes.io/projected/95a07519-99bf-47ac-b236-c642491d5648-kube-api-access-s5r4v\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:16.462279 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462216    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysctl-d\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.462279 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462257    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-systemd\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.462380 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462286    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-cni-bin\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.462380 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462322    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-sys-fs\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.462380 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462348    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-kubelet-config\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462381    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77q9p\" (UniqueName: \"kubernetes.io/projected/49a42451-a2db-456c-93ed-e0b932f13751-kube-api-access-77q9p\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462392    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-cni-bin\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462406    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvgls\" (UniqueName: \"kubernetes.io/projected/971e53b3-ff90-4be0-bf64-9c1013099b9d-kube-api-access-hvgls\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462433    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462436    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysctl-d\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462459    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462464    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-systemd\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462484    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-host\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462506    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysctl-conf\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462529    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-kubelet\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462551    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-env-overrides\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462563    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovnkube-config\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.462579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462573    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cnibin\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462606    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-hosts-file\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462651    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462681    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-cni-bin\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462692    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-kubelet-config\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462707    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462742    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysctl-conf\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.462767    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462778    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-host\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462805    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-kubelet\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.462839    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs podName:95a07519-99bf-47ac-b236-c642491d5648 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:16.96282604 +0000 UTC m=+3.142553500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs") pod "network-metrics-daemon-k6rks" (UID: "95a07519-99bf-47ac-b236-c642491d5648") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462857    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-system-cni-dir\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462885    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m9gn\" (UniqueName: \"kubernetes.io/projected/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-kube-api-access-9m9gn\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462902    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49a42451-a2db-456c-93ed-e0b932f13751-etc-tuned\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462917    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-multus-certs\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462932    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58r88\" (UniqueName: \"kubernetes.io/projected/eb190b5e-126c-4d02-9c3a-35dc5c10d521-kube-api-access-58r88\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462963    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.463208 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462979    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462992    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462768    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463010    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-hostroot\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463025    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-slash\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463045    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-etc-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463070    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-serviceca\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463084    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-host\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463100    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2f9l\" (UniqueName: \"kubernetes.io/projected/5f8769fd-d1f7-4e86-b367-e5cc0e707174-kube-api-access-j2f9l\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.462656    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-sys-fs\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463115    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-kubelet\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463141    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-run-ovn-kubernetes\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463150    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cnibin\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463161    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovn-node-metrics-cert\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463176    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-multus-certs\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463186    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-hosts-file\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463179    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovnkube-script-lib\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464022 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463205    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-kubelet\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463211    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463227    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-modprobe-d\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463242    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysconfig\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463233    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-env-overrides\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463266    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-run-ovn-kubernetes\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463318    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-cni-bin\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463260    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-lib-modules\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463337    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-lib-modules\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463361    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-sys\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463385    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-systemd-units\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463413    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysconfig\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463410    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-systemd\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463434    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-ovn\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463447    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-systemd\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463449    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-socket-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463462    2544 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463483    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a42451-a2db-456c-93ed-e0b932f13751-tmp\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.464827 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463507    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-cni-netd\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463519    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-socket-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.463243    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463531    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-etc-kubernetes\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463567    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-ovn\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463567    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovnkube-script-lib\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463573    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-run-netns\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463602    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-sys\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463600    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-registration-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463643    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-dbus\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463664    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-system-cni-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463665    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-registration-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463669    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-etc-kubernetes\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463679    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-cni-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463708    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-run-netns\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463780    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-dbus\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463829    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-cni-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.465698 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463856    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-system-cni-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.463865    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret podName:cd7651fe-9b08-41ab-b4e4-5df2fbee97ee nodeName:}" failed. No retries permitted until 2026-10-30 02:54:16.963847804 +0000 UTC m=+3.143575256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret") pod "global-pull-secret-syncer-ffnvh" (UID: "cd7651fe-9b08-41ab-b4e4-5df2fbee97ee") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463885    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-cnibin\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463890    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-cni-netd\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463911    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-var-lib-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463918    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-etc-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463937    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463963    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-run\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463966    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-hostroot\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463968    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463987    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55b4z\" (UniqueName: \"kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z\") pod \"network-check-target-xg9mx\" (UID: \"23d4d339-696a-4b94-b34f-19852eb3cbc2\") " pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464021    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-socket-dir-parent\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464045    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-daemon-config\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464067    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-tmp-dir\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464087    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464108    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.466461 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464131    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-netns\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464150    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464171    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-device-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464193    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9r6z\" (UniqueName: \"kubernetes.io/projected/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-kube-api-access-p9r6z\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464214    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2811f6f-7a04-455e-953c-18c1018672d8-konnectivity-ca\") pod \"konnectivity-agent-x5wjw\" (UID: \"e2811f6f-7a04-455e-953c-18c1018672d8\") " pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464240    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb190b5e-126c-4d02-9c3a-35dc5c10d521-iptables-alerter-script\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464243    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-cnibin\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464260    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb190b5e-126c-4d02-9c3a-35dc5c10d521-host-slash\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464283    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-var-lib-kubelet\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464288    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-var-lib-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464306    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-k8s-cni-cncf-io\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464309    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-serviceca\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464328    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-cni-multus\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464351    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-conf-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464370    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-node-log\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464378    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464380    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-host\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.467286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463940    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-system-cni-dir\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464397    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2811f6f-7a04-455e-953c-18c1018672d8-agent-certs\") pod \"konnectivity-agent-x5wjw\" (UID: \"e2811f6f-7a04-455e-953c-18c1018672d8\") " pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464434    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-kubernetes\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464458    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f8769fd-d1f7-4e86-b367-e5cc0e707174-cni-binary-copy\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464488    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzw7\" (UniqueName: \"kubernetes.io/projected/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-kube-api-access-nhzw7\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464512    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-log-socket\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.463388    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-modprobe-d\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464588    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-kubernetes\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464590    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-log-socket\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.464666    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-run\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.465305    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f8769fd-d1f7-4e86-b367-e5cc0e707174-cni-binary-copy\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.465362    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-systemd-units\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.465418    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-device-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.465455    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-slash\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.465503    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-socket-dir-parent\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.465971    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-daemon-config\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.466271    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-var-lib-kubelet\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.466629    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-cni-multus\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.468097 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.466668    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-k8s-cni-cncf-io\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.466733    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb190b5e-126c-4d02-9c3a-35dc5c10d521-host-slash\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.466791    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-conf-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.466831    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-node-log\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.467282    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-tmp-dir\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.467315    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49a42451-a2db-456c-93ed-e0b932f13751-etc-tuned\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.467351    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-netns\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.467383    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.467483    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.467690    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2811f6f-7a04-455e-953c-18c1018672d8-agent-certs\") pod \"konnectivity-agent-x5wjw\" (UID: \"e2811f6f-7a04-455e-953c-18c1018672d8\") " pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.467822    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a42451-a2db-456c-93ed-e0b932f13751-tmp\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.467912    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovn-node-metrics-cert\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.467949    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2811f6f-7a04-455e-953c-18c1018672d8-konnectivity-ca\") pod \"konnectivity-agent-x5wjw\" (UID: \"e2811f6f-7a04-455e-953c-18c1018672d8\") " pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.468391    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.468865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.468425    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb190b5e-126c-4d02-9c3a-35dc5c10d521-iptables-alerter-script\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:16.498487 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.498454    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvgls\" (UniqueName: \"kubernetes.io/projected/971e53b3-ff90-4be0-bf64-9c1013099b9d-kube-api-access-hvgls\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.501878 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.501670    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:16.501878 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.501696    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:16.501878 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.501709    2544 projected.go:196] Error preparing data for projected volume kube-api-access-55b4z for pod openshift-network-diagnostics/network-check-target-xg9mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:16.501878 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.501770    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z podName:23d4d339-696a-4b94-b34f-19852eb3cbc2 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:17.001752001 +0000 UTC m=+3.181479449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-55b4z" (UniqueName: "kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z") pod "network-check-target-xg9mx" (UID: "23d4d339-696a-4b94-b34f-19852eb3cbc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:16.506998 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.504460    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m9gn\" (UniqueName: \"kubernetes.io/projected/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-kube-api-access-9m9gn\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:16.506998 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.505415    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzw7\" (UniqueName: \"kubernetes.io/projected/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-kube-api-access-nhzw7\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:16.506998 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.506694    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5r4v\" (UniqueName: \"kubernetes.io/projected/95a07519-99bf-47ac-b236-c642491d5648-kube-api-access-s5r4v\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:16.506998 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.506836    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdqh\" (UniqueName: \"kubernetes.io/projected/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-kube-api-access-vgdqh\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.507741 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.507236    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58r88\" (UniqueName: \"kubernetes.io/projected/eb190b5e-126c-4d02-9c3a-35dc5c10d521-kube-api-access-58r88\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:16.509253 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.509228    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2f9l\" (UniqueName: \"kubernetes.io/projected/5f8769fd-d1f7-4e86-b367-e5cc0e707174-kube-api-access-j2f9l\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.513546 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.513516    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9r6z\" (UniqueName: \"kubernetes.io/projected/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-kube-api-access-p9r6z\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.513889 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.513869    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77q9p\" (UniqueName: \"kubernetes.io/projected/49a42451-a2db-456c-93ed-e0b932f13751-kube-api-access-77q9p\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.577035 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.576778    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:16.586368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.586338    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:16.589008 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:16.588979    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2811f6f_7a04_455e_953c_18c1018672d8.slice/crio-8bc7d9bf3be6c7bf2601c610170b8db20e511211e95acdb9077f35ed2b6c5a2c WatchSource:0}: Error finding container 8bc7d9bf3be6c7bf2601c610170b8db20e511211e95acdb9077f35ed2b6c5a2c: Status 404 returned error can't find the container with id 8bc7d9bf3be6c7bf2601c610170b8db20e511211e95acdb9077f35ed2b6c5a2c
Oct 30 02:54:16.594502 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.594476    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:16.595954 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:16.595928    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8769fd_d1f7_4e86_b367_e5cc0e707174.slice/crio-40b04b27e2803713a170055e19d04f89b19fe802609728b89711fab94f7a71ef WatchSource:0}: Error finding container 40b04b27e2803713a170055e19d04f89b19fe802609728b89711fab94f7a71ef: Status 404 returned error can't find the container with id 40b04b27e2803713a170055e19d04f89b19fe802609728b89711fab94f7a71ef
Oct 30 02:54:16.603094 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.603056    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:16.605573 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:16.605546    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef4e0a8_2d45_4d38_b01d_806fdb0e4ff5.slice/crio-3640247c6527041991663673bdc0886b288f5d05f09e2c4eee4c87725c12f5d0 WatchSource:0}: Error finding container 3640247c6527041991663673bdc0886b288f5d05f09e2c4eee4c87725c12f5d0: Status 404 returned error can't find the container with id 3640247c6527041991663673bdc0886b288f5d05f09e2c4eee4c87725c12f5d0
Oct 30 02:54:16.614115 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:16.614090    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb190b5e_126c_4d02_9c3a_35dc5c10d521.slice/crio-6ce27297772f2101afb0562eb3ad266d8d0ad36cfa4e7e1a8dfa2bf0725585b2 WatchSource:0}: Error finding container 6ce27297772f2101afb0562eb3ad266d8d0ad36cfa4e7e1a8dfa2bf0725585b2: Status 404 returned error can't find the container with id 6ce27297772f2101afb0562eb3ad266d8d0ad36cfa4e7e1a8dfa2bf0725585b2
Oct 30 02:54:16.646188 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.646106    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:16.648206 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.648183    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:16.654262 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.654118    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:16.658049 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:16.657943    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971e53b3_ff90_4be0_bf64_9c1013099b9d.slice/crio-2d04fc5c3be2d62b7033ea66d8a484795d952ab6e24c0103eb44231dd40f84dd WatchSource:0}: Error finding container 2d04fc5c3be2d62b7033ea66d8a484795d952ab6e24c0103eb44231dd40f84dd: Status 404 returned error can't find the container with id 2d04fc5c3be2d62b7033ea66d8a484795d952ab6e24c0103eb44231dd40f84dd
Oct 30 02:54:16.659243 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:16.658774    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb32bcb9_2633_49f9_9e36_b378cf3a7b09.slice/crio-576d2749d1f7b505d76f101f7652276ce4d2e1428828ff65b8d7822a138c8010 WatchSource:0}: Error finding container 576d2749d1f7b505d76f101f7652276ce4d2e1428828ff65b8d7822a138c8010: Status 404 returned error can't find the container with id 576d2749d1f7b505d76f101f7652276ce4d2e1428828ff65b8d7822a138c8010
Oct 30 02:54:16.659243 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.658867    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:16.662693 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.662517    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:16.664182 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:16.664158    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a42451_a2db_456c_93ed_e0b932f13751.slice/crio-b5012e56b4cfbd5cef5e3523041aaf9ddc499ea37c9c521dc38bba3c070da2bb WatchSource:0}: Error finding container b5012e56b4cfbd5cef5e3523041aaf9ddc499ea37c9c521dc38bba3c070da2bb: Status 404 returned error can't find the container with id b5012e56b4cfbd5cef5e3523041aaf9ddc499ea37c9c521dc38bba3c070da2bb
Oct 30 02:54:16.967084 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.967052    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:16.967221 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:16.967125    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:16.967268 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.967242    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:16.967325 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.967302    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs podName:95a07519-99bf-47ac-b236-c642491d5648 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:17.967284788 +0000 UTC m=+4.147012213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs") pod "network-metrics-daemon-k6rks" (UID: "95a07519-99bf-47ac-b236-c642491d5648") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:16.967677 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.967656    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:16.967779 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:16.967726    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret podName:cd7651fe-9b08-41ab-b4e4-5df2fbee97ee nodeName:}" failed. No retries permitted until 2026-10-30 02:54:17.967700305 +0000 UTC m=+4.147427730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret") pod "global-pull-secret-syncer-ffnvh" (UID: "cd7651fe-9b08-41ab-b4e4-5df2fbee97ee") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:17.069008 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.068374    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55b4z\" (UniqueName: \"kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z\") pod \"network-check-target-xg9mx\" (UID: \"23d4d339-696a-4b94-b34f-19852eb3cbc2\") " pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:17.069008 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:17.068869    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:17.069008 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:17.068895    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:17.069008 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:17.068910    2544 projected.go:196] Error preparing data for projected volume kube-api-access-55b4z for pod openshift-network-diagnostics/network-check-target-xg9mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:17.069008 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:17.068968    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z podName:23d4d339-696a-4b94-b34f-19852eb3cbc2 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:18.068949273 +0000 UTC m=+4.248676716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-55b4z" (UniqueName: "kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z") pod "network-check-target-xg9mx" (UID: "23d4d339-696a-4b94-b34f-19852eb3cbc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:17.320236 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.320104    2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:49:16 +0000 UTC" deadline="2028-04-13 15:13:26.947301104 +0000 UTC"
Oct 30 02:54:17.320236 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.320153    2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12756h19m9.627151424s"
Oct 30 02:54:17.405642 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.405461    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:17.410575 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:17.405869    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:17.428590 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.428556    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal" event={"ID":"7470c725c1ecc41c5e63c668787a572e","Type":"ContainerStarted","Data":"5bff568864da222fd48d864706d480b3853d6c2efe10af29f2e8736084712fab"}
Oct 30 02:54:17.432581 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.431958    2544 generic.go:356] "Generic (PLEG): container finished" podID="d6b355a5b292f9d5c5019135ffc6b2ed" containerID="b90ed3c96f4d231839ea9e4762c63140fb5c0ad01203fb9d4e9683faf6b6ac87" exitCode=0
Oct 30 02:54:17.432581 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.432031    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal" event={"ID":"d6b355a5b292f9d5c5019135ffc6b2ed","Type":"ContainerDied","Data":"b90ed3c96f4d231839ea9e4762c63140fb5c0ad01203fb9d4e9683faf6b6ac87"}
Oct 30 02:54:17.442783 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.442725    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlhb7" event={"ID":"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6","Type":"ContainerStarted","Data":"400509b082b25f36fe9604165f78b1ab77eda3bc29553f12547af6a9bb2e9ebd"}
Oct 30 02:54:17.462078 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.462051    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t" event={"ID":"8ddbeb45-7185-4bc0-91de-dd2619eda1d4","Type":"ContainerStarted","Data":"9ab991f730660e0a52c790a171b7db61a7d4bf7514ac35a2ebee23d2416666ae"}
Oct 30 02:54:17.485184 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.485155    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-stg5z" event={"ID":"eb32bcb9-2633-49f9-9e36-b378cf3a7b09","Type":"ContainerStarted","Data":"576d2749d1f7b505d76f101f7652276ce4d2e1428828ff65b8d7822a138c8010"}
Oct 30 02:54:17.520330 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.520226    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244kd" event={"ID":"971e53b3-ff90-4be0-bf64-9c1013099b9d","Type":"ContainerStarted","Data":"2d04fc5c3be2d62b7033ea66d8a484795d952ab6e24c0103eb44231dd40f84dd"}
Oct 30 02:54:17.522352 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.522192    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal" podStartSLOduration=2.522174016 podStartE2EDuration="2.522174016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:17.475379223 +0000 UTC m=+3.655106674" watchObservedRunningTime="2026-10-30 02:54:17.522174016 +0000 UTC m=+3.701901468"
Oct 30 02:54:17.522352 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.522279    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6th7l" event={"ID":"eb190b5e-126c-4d02-9c3a-35dc5c10d521","Type":"ContainerStarted","Data":"6ce27297772f2101afb0562eb3ad266d8d0ad36cfa4e7e1a8dfa2bf0725585b2"}
Oct 30 02:54:17.532583 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.532529    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k6c24" event={"ID":"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5","Type":"ContainerStarted","Data":"3640247c6527041991663673bdc0886b288f5d05f09e2c4eee4c87725c12f5d0"}
Oct 30 02:54:17.547656 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.547628    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l6rrk" event={"ID":"49a42451-a2db-456c-93ed-e0b932f13751","Type":"ContainerStarted","Data":"b5012e56b4cfbd5cef5e3523041aaf9ddc499ea37c9c521dc38bba3c070da2bb"}
Oct 30 02:54:17.553754 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.553686    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fbpcs" event={"ID":"5f8769fd-d1f7-4e86-b367-e5cc0e707174","Type":"ContainerStarted","Data":"40b04b27e2803713a170055e19d04f89b19fe802609728b89711fab94f7a71ef"}
Oct 30 02:54:17.555754 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.555720    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x5wjw" event={"ID":"e2811f6f-7a04-455e-953c-18c1018672d8","Type":"ContainerStarted","Data":"8bc7d9bf3be6c7bf2601c610170b8db20e511211e95acdb9077f35ed2b6c5a2c"}
Oct 30 02:54:17.977534 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.976788    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:17.977534 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:17.976844    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:17.977534 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:17.976961    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:17.977534 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:17.977021    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret podName:cd7651fe-9b08-41ab-b4e4-5df2fbee97ee nodeName:}" failed. No retries permitted until 2026-10-30 02:54:19.977002704 +0000 UTC m=+6.156730133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret") pod "global-pull-secret-syncer-ffnvh" (UID: "cd7651fe-9b08-41ab-b4e4-5df2fbee97ee") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:17.977534 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:17.977391    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:17.977534 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:17.977437    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs podName:95a07519-99bf-47ac-b236-c642491d5648 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:19.977423766 +0000 UTC m=+6.157151196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs") pod "network-metrics-daemon-k6rks" (UID: "95a07519-99bf-47ac-b236-c642491d5648") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:18.077361 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:18.077327    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55b4z\" (UniqueName: \"kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z\") pod \"network-check-target-xg9mx\" (UID: \"23d4d339-696a-4b94-b34f-19852eb3cbc2\") " pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:18.077526 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:18.077504    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:18.077526 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:18.077527    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:18.077660 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:18.077540    2544 projected.go:196] Error preparing data for projected volume kube-api-access-55b4z for pod openshift-network-diagnostics/network-check-target-xg9mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:18.077660 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:18.077607    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z podName:23d4d339-696a-4b94-b34f-19852eb3cbc2 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:20.077589117 +0000 UTC m=+6.257316542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-55b4z" (UniqueName: "kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z") pod "network-check-target-xg9mx" (UID: "23d4d339-696a-4b94-b34f-19852eb3cbc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:18.320505 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:18.320439    2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:49:16 +0000 UTC" deadline="2028-06-26 22:02:12.456009459 +0000 UTC"
Oct 30 02:54:18.320505 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:18.320500    2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14539h7m54.135513647s"
Oct 30 02:54:18.407375 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:18.407345    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:18.407523 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:18.407478    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:18.409081 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:18.407894    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:18.409081 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:18.407978    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:18.581929 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:18.581761    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal" event={"ID":"d6b355a5b292f9d5c5019135ffc6b2ed","Type":"ContainerStarted","Data":"f812732fecc6085b8f971f812c4cd77f69b15bce7d5bfca1c18292366fc78877"}
Oct 30 02:54:18.627734 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:18.627544    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal" podStartSLOduration=3.627523189 podStartE2EDuration="3.627523189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:18.62642826 +0000 UTC m=+4.806155709" watchObservedRunningTime="2026-10-30 02:54:18.627523189 +0000 UTC m=+4.807250639"
Oct 30 02:54:19.405076 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:19.404586    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:19.405076 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:19.404741    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:19.589657 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:19.588423    2544 generic.go:356] "Generic (PLEG): container finished" podID="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" containerID="674dd5146b90b388d1280423087eb3b7c1788ce50e0fc957bec9e1188d807c0f" exitCode=0
Oct 30 02:54:19.589657 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:19.589421    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlhb7" event={"ID":"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6","Type":"ContainerDied","Data":"674dd5146b90b388d1280423087eb3b7c1788ce50e0fc957bec9e1188d807c0f"}
Oct 30 02:54:19.991530 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:19.990857    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:19.991530 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:19.990912    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:19.991530 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:19.991066    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:19.991530 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:19.991127    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret podName:cd7651fe-9b08-41ab-b4e4-5df2fbee97ee nodeName:}" failed. No retries permitted until 2026-10-30 02:54:23.991110389 +0000 UTC m=+10.170837830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret") pod "global-pull-secret-syncer-ffnvh" (UID: "cd7651fe-9b08-41ab-b4e4-5df2fbee97ee") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:19.991530 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:19.991433    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:19.991530 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:19.991465    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs podName:95a07519-99bf-47ac-b236-c642491d5648 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:23.991456219 +0000 UTC m=+10.171183657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs") pod "network-metrics-daemon-k6rks" (UID: "95a07519-99bf-47ac-b236-c642491d5648") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:20.091950 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:20.091890    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55b4z\" (UniqueName: \"kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z\") pod \"network-check-target-xg9mx\" (UID: \"23d4d339-696a-4b94-b34f-19852eb3cbc2\") " pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:20.092129 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:20.092046    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:20.092129 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:20.092062    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:20.092129 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:20.092084    2544 projected.go:196] Error preparing data for projected volume kube-api-access-55b4z for pod openshift-network-diagnostics/network-check-target-xg9mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:20.092221 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:20.092141    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z podName:23d4d339-696a-4b94-b34f-19852eb3cbc2 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:24.092122689 +0000 UTC m=+10.271850129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-55b4z" (UniqueName: "kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z") pod "network-check-target-xg9mx" (UID: "23d4d339-696a-4b94-b34f-19852eb3cbc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:20.408807 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:20.408776    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:20.409242 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:20.408895    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:20.409242 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:20.409171    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:20.409338 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:20.409242    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:21.405505 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:21.405476    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:21.405669 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:21.405630    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:21.604106 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:21.604029    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-stg5z" event={"ID":"eb32bcb9-2633-49f9-9e36-b378cf3a7b09","Type":"ContainerStarted","Data":"31f2c9014d34c7ccdd68014b830e563728d968138b7fe585e1bb904c7f89293c"}
Oct 30 02:54:21.607689 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:21.607262    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x5wjw" event={"ID":"e2811f6f-7a04-455e-953c-18c1018672d8","Type":"ContainerStarted","Data":"d6d37efa3abbad59414ccabad2ad474317508115858b80e2ef4218dde40273d3"}
Oct 30 02:54:21.651536 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:21.650839    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-stg5z" podStartSLOduration=3.116181756 podStartE2EDuration="7.650822007s" totalImagesPullingTime="4.534640251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:21.650174779 +0000 UTC m=+7.829902237" watchObservedRunningTime="2026-10-30 02:54:21.650822007 +0000 UTC m=+7.830549455"
Oct 30 02:54:21.704184 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:21.704139    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x5wjw" podStartSLOduration=3.960198482 podStartE2EDuration="7.704123847s" totalImagesPullingTime="3.743925365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:21.703465827 +0000 UTC m=+7.883193277" watchObservedRunningTime="2026-10-30 02:54:21.704123847 +0000 UTC m=+7.883851294"
Oct 30 02:54:22.407767 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:22.407731    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:22.407906 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:22.407865    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:22.408240 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:22.408223    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:22.408338 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:22.408317    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:22.613703 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:22.612982    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t" event={"ID":"8ddbeb45-7185-4bc0-91de-dd2619eda1d4","Type":"ContainerStarted","Data":"4b0d8f3f2f4617bec4aeceac7c774cf6e02798c77e99475410df7538cf74cc31"}
Oct 30 02:54:23.404868 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:23.404821    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:23.405030 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:23.404968    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:24.025919 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.025877    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:24.026419 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.025983    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:24.026419 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:24.026112    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:24.026419 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:24.026178    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs podName:95a07519-99bf-47ac-b236-c642491d5648 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:32.026160771 +0000 UTC m=+18.205888220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs") pod "network-metrics-daemon-k6rks" (UID: "95a07519-99bf-47ac-b236-c642491d5648") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:24.026603 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:24.026577    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:24.026675 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:24.026637    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret podName:cd7651fe-9b08-41ab-b4e4-5df2fbee97ee nodeName:}" failed. No retries permitted until 2026-10-30 02:54:32.026606876 +0000 UTC m=+18.206334301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret") pod "global-pull-secret-syncer-ffnvh" (UID: "cd7651fe-9b08-41ab-b4e4-5df2fbee97ee") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:24.115922 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.115434    2544 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:54:24.126688 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.126355    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55b4z\" (UniqueName: \"kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z\") pod \"network-check-target-xg9mx\" (UID: \"23d4d339-696a-4b94-b34f-19852eb3cbc2\") " pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:24.126688 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:24.126542    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:24.126688 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:24.126577    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:24.126688 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:24.126590    2544 projected.go:196] Error preparing data for projected volume kube-api-access-55b4z for pod openshift-network-diagnostics/network-check-target-xg9mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:24.126688 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:24.126659    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z podName:23d4d339-696a-4b94-b34f-19852eb3cbc2 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:32.12663999 +0000 UTC m=+18.306367430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-55b4z" (UniqueName: "kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z") pod "network-check-target-xg9mx" (UID: "23d4d339-696a-4b94-b34f-19852eb3cbc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:24.317312 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.317215    2544 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:54:24.115456115Z","UUID":"47cabdc6-f74e-4f3b-9b8a-50e8efdbab96","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:54:24.323213 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.320473    2544 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:54:24.323213 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.320507    2544 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:54:24.406849 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.406805    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:24.407012 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:24.406909    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:24.407012 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.406960    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:24.407109 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:24.407041    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:24.455408 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.453863    2544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:24.455408 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.454418    2544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:24.619909 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.619828    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k6c24" event={"ID":"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5","Type":"ContainerStarted","Data":"aaf2a36b9016abc14302285a3a0b2ecb2316d80a97f794b8aee8967501b9bcc0"}
Oct 30 02:54:24.622788 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.622758    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fbpcs" event={"ID":"5f8769fd-d1f7-4e86-b367-e5cc0e707174","Type":"ContainerStarted","Data":"b072f9ea03ffd3455c59b3709c86db4f645fe5f58dd72fc53234a906c01c2b97"}
Oct 30 02:54:24.624870 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.624839    2544 generic.go:356] "Generic (PLEG): container finished" podID="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" containerID="d918cc83812d35382645a134e8e022f016e15a4f42cb43ebbe3cd5a10b422174" exitCode=0
Oct 30 02:54:24.624970 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.624915    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlhb7" event={"ID":"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6","Type":"ContainerDied","Data":"d918cc83812d35382645a134e8e022f016e15a4f42cb43ebbe3cd5a10b422174"}
Oct 30 02:54:24.628752 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.628729    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t" event={"ID":"8ddbeb45-7185-4bc0-91de-dd2619eda1d4","Type":"ContainerStarted","Data":"dfb48365b4b456ca459807b17483bee2aabc95f3ce0ce2922a1eccea7f04f939"}
Oct 30 02:54:24.628843 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.628785    2544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:24.629168 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.629148    2544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:24.710443 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.710387    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-k6c24" podStartSLOduration=3.010851664 podStartE2EDuration="10.710368299s" totalImagesPullingTime="7.699516635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:24.665922808 +0000 UTC m=+10.845650266" watchObservedRunningTime="2026-10-30 02:54:24.710368299 +0000 UTC m=+10.890095748"
Oct 30 02:54:24.807924 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:24.807720    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-fbpcs" podStartSLOduration=3.143842616 podStartE2EDuration="10.807697902s" totalImagesPullingTime="7.663855286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:24.757709179 +0000 UTC m=+10.937436625" watchObservedRunningTime="2026-10-30 02:54:24.807697902 +0000 UTC m=+10.987425352"
Oct 30 02:54:25.405303 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:25.405272    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:25.405743 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:25.405405    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:25.633750 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:25.633642    2544 generic.go:356] "Generic (PLEG): container finished" podID="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" containerID="c095b04604c4f8dcf61cde9a94d9a3c49a320469378e19e4078fb47d552744c4" exitCode=0
Oct 30 02:54:25.633750 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:25.633695    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlhb7" event={"ID":"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6","Type":"ContainerDied","Data":"c095b04604c4f8dcf61cde9a94d9a3c49a320469378e19e4078fb47d552744c4"}
Oct 30 02:54:25.638725 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:25.638695    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t" event={"ID":"8ddbeb45-7185-4bc0-91de-dd2619eda1d4","Type":"ContainerStarted","Data":"6fb78d2d3cd06b04cc6adf63e1263c3edf028012e1d6bd8a8a6af094b2c2b39b"}
Oct 30 02:54:25.640738 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:25.640688    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6th7l" event={"ID":"eb190b5e-126c-4d02-9c3a-35dc5c10d521","Type":"ContainerStarted","Data":"91af54acf4064496265c6b0875eee82a2cfd9c39f744a24b41ada3800049e66f"}
Oct 30 02:54:25.732109 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:25.732011    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6th7l" podStartSLOduration=4.271167751 podStartE2EDuration="11.731992539s" totalImagesPullingTime="7.460824788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:25.731774406 +0000 UTC m=+11.911501855" watchObservedRunningTime="2026-10-30 02:54:25.731992539 +0000 UTC m=+11.911719986"
Oct 30 02:54:25.780972 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:25.780911    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t" podStartSLOduration=3.85033393 podStartE2EDuration="11.780891944s" totalImagesPullingTime="7.930558014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:25.780033714 +0000 UTC m=+11.959761186" watchObservedRunningTime="2026-10-30 02:54:25.780891944 +0000 UTC m=+11.960619392"
Oct 30 02:54:26.405311 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:26.404839    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:26.405311 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:26.404847    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:26.405311 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:26.404974    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:26.405311 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:26.405036    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:26.644477 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:26.644443    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l6rrk" event={"ID":"49a42451-a2db-456c-93ed-e0b932f13751","Type":"ContainerStarted","Data":"e4d3018f2b8d29dc0c3f5b75d96940310a1c7b96a0dc066e83967af42da232b4"}
Oct 30 02:54:26.647271 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:26.647233    2544 generic.go:356] "Generic (PLEG): container finished" podID="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" containerID="350af7331f91a2b3ed306bfba8d0fe6c80dcddd64bff5a64d10ddbe8d1b9a360" exitCode=0
Oct 30 02:54:26.647361 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:26.647299    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlhb7" event={"ID":"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6","Type":"ContainerDied","Data":"350af7331f91a2b3ed306bfba8d0fe6c80dcddd64bff5a64d10ddbe8d1b9a360"}
Oct 30 02:54:26.695403 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:26.695303    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l6rrk" podStartSLOduration=2.8464594869999997 podStartE2EDuration="12.695282818s" totalImagesPullingTime="9.848823331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:26.69466704 +0000 UTC m=+12.874394498" watchObservedRunningTime="2026-10-30 02:54:26.695282818 +0000 UTC m=+12.875010267"
Oct 30 02:54:27.404691 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:27.404656    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:27.404854 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:27.404808    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:28.405203 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:28.405167    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:28.405203 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:28.405200    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:28.405967 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:28.405290    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:28.405967 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:28.405414    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:28.654741 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:28.654709    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlhb7" event={"ID":"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6","Type":"ContainerStarted","Data":"0818a7040d46fd92e3aae3a2064f6276b29b036c959897a5494058dac9d22dca"}
Oct 30 02:54:29.404895 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:29.404662    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:29.405062 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:29.405000    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:29.660160 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:29.660038    2544 generic.go:356] "Generic (PLEG): container finished" podID="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" containerID="0818a7040d46fd92e3aae3a2064f6276b29b036c959897a5494058dac9d22dca" exitCode=0
Oct 30 02:54:29.660160 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:29.660096    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlhb7" event={"ID":"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6","Type":"ContainerDied","Data":"0818a7040d46fd92e3aae3a2064f6276b29b036c959897a5494058dac9d22dca"}
Oct 30 02:54:30.404660 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:30.404605    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:30.404831 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:30.404730    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:30.404831 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:30.404782    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:30.404954 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:30.404872    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:30.664536 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:30.664508    2544 generic.go:356] "Generic (PLEG): container finished" podID="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" containerID="86cf573a37834005d8285b8fbf05ed4917fde5d9052403ba70d390fd9eedc8bb" exitCode=0
Oct 30 02:54:30.664835 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:30.664555    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlhb7" event={"ID":"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6","Type":"ContainerDied","Data":"86cf573a37834005d8285b8fbf05ed4917fde5d9052403ba70d390fd9eedc8bb"}
Oct 30 02:54:31.404941 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:31.404715    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:31.405088 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:31.404982    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:31.670079 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:31.669988    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hlhb7" event={"ID":"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6","Type":"ContainerStarted","Data":"0f5936b2d3c2cc1a6af20769989e1abcb24d383d999f1d8be93a5ef645805a94"}
Oct 30 02:54:31.672742 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:31.672723    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 02:54:31.673014 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:31.672996    2544 generic.go:356] "Generic (PLEG): container finished" podID="971e53b3-ff90-4be0-bf64-9c1013099b9d" containerID="dd0fa76e37e94099d86c7cade33ec3c32f2c4f6c90f4551184b90baacff9e67c" exitCode=1
Oct 30 02:54:31.673060 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:31.673029    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244kd" event={"ID":"971e53b3-ff90-4be0-bf64-9c1013099b9d","Type":"ContainerStarted","Data":"e8ce08b5a5fe40b0787e6653386eea90f82ea6e9486f10fd43a3c15693c12dc6"}
Oct 30 02:54:31.673089 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:31.673079    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244kd" event={"ID":"971e53b3-ff90-4be0-bf64-9c1013099b9d","Type":"ContainerStarted","Data":"6ef7dcd5dbdc9159df1f16f145c6487fdb2325eddde6befc36c2ec57b88a5981"}
Oct 30 02:54:31.673135 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:31.673109    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244kd" event={"ID":"971e53b3-ff90-4be0-bf64-9c1013099b9d","Type":"ContainerStarted","Data":"f97d4a8da152e131f0d8432c3d50e548c7c142121e079d15a9ed9ded6ad04e9c"}
Oct 30 02:54:31.673163 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:31.673137    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244kd" event={"ID":"971e53b3-ff90-4be0-bf64-9c1013099b9d","Type":"ContainerStarted","Data":"7629fbd95bb43d0081dae3336678b371c2f340d4d2b20d0efeb97a4554bcab30"}
Oct 30 02:54:31.673194 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:31.673162    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244kd" event={"ID":"971e53b3-ff90-4be0-bf64-9c1013099b9d","Type":"ContainerDied","Data":"dd0fa76e37e94099d86c7cade33ec3c32f2c4f6c90f4551184b90baacff9e67c"}
Oct 30 02:54:31.673194 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:31.673190    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244kd" event={"ID":"971e53b3-ff90-4be0-bf64-9c1013099b9d","Type":"ContainerStarted","Data":"ee4cb3c9b664e713caa2aebab764058e63a1f8234fbbbc3bf56232da7792562b"}
Oct 30 02:54:32.093077 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:32.092999    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:32.093229 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:32.093079    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:32.093229 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:32.093134    2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:32.093229 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:32.093156    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:32.093229 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:32.093211    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret podName:cd7651fe-9b08-41ab-b4e4-5df2fbee97ee nodeName:}" failed. No retries permitted until 2026-10-30 02:54:48.093191706 +0000 UTC m=+34.272919156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret") pod "global-pull-secret-syncer-ffnvh" (UID: "cd7651fe-9b08-41ab-b4e4-5df2fbee97ee") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:32.093378 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:32.093233    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs podName:95a07519-99bf-47ac-b236-c642491d5648 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:48.093223682 +0000 UTC m=+34.272951124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs") pod "network-metrics-daemon-k6rks" (UID: "95a07519-99bf-47ac-b236-c642491d5648") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:32.193783 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:32.193750    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55b4z\" (UniqueName: \"kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z\") pod \"network-check-target-xg9mx\" (UID: \"23d4d339-696a-4b94-b34f-19852eb3cbc2\") " pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:32.193919 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:32.193880    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:32.193919 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:32.193894    2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:32.193919 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:32.193907    2544 projected.go:196] Error preparing data for projected volume kube-api-access-55b4z for pod openshift-network-diagnostics/network-check-target-xg9mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:32.194069 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:32.193962    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z podName:23d4d339-696a-4b94-b34f-19852eb3cbc2 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:48.193943639 +0000 UTC m=+34.373671078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-55b4z" (UniqueName: "kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z") pod "network-check-target-xg9mx" (UID: "23d4d339-696a-4b94-b34f-19852eb3cbc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:32.404580 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:32.404507    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:32.404721 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:32.404648    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:32.404721 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:32.404700    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:32.404814 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:32.404793    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:33.405140 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:33.405045    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:33.405582 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:33.405189    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:33.680202 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:33.680131    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 02:54:33.680449 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:33.680429    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244kd" event={"ID":"971e53b3-ff90-4be0-bf64-9c1013099b9d","Type":"ContainerStarted","Data":"f061e6f27ec65b19f8608e1081dc2efa7a31d4f00342fd196c31146a35f8094b"}
Oct 30 02:54:34.405302 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:34.405267    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:34.405747 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:34.405375    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:34.405747 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:34.405466    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:34.405747 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:34.405575    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:35.405537 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:35.405499    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:35.406182 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:35.405633    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:35.688741 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:35.688665    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 02:54:35.689048 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:35.689023    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244kd" event={"ID":"971e53b3-ff90-4be0-bf64-9c1013099b9d","Type":"ContainerStarted","Data":"da573a3ea15436ebb72cfc0e1d811a65bcda6421b87fad826470196d4809237c"}
Oct 30 02:54:35.689337 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:35.689317    2544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:35.689426 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:35.689379    2544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:35.689490 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:35.689463    2544 scope.go:122] "RemoveContainer" containerID="dd0fa76e37e94099d86c7cade33ec3c32f2c4f6c90f4551184b90baacff9e67c"
Oct 30 02:54:35.706011 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:35.705990    2544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:35.799052 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:35.798993    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hlhb7" podStartSLOduration=11.386728261 podStartE2EDuration="21.798978323s" totalImagesPullingTime="10.412250062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:31.751227417 +0000 UTC m=+17.930954865" watchObservedRunningTime="2026-10-30 02:54:35.798978323 +0000 UTC m=+21.978705769"
Oct 30 02:54:36.405355 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:36.405296    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:36.405355 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:36.405344    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:36.405591 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:36.405444    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:36.406036 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:36.405649    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:36.697582 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:36.697499    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 02:54:36.697864 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:36.697841    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244kd" event={"ID":"971e53b3-ff90-4be0-bf64-9c1013099b9d","Type":"ContainerStarted","Data":"b92ac81a9ba264586206833a8b7ec47c68921d61a8b9a862b83ebc8feb84bbe9"}
Oct 30 02:54:36.698031 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:36.698018    2544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:36.713758 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:36.713728    2544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:36.807375 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:36.807325    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-244kd" podStartSLOduration=8.824753246 podStartE2EDuration="22.807313058s" totalImagesPullingTime="13.982559812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:36.807001626 +0000 UTC m=+22.986729072" watchObservedRunningTime="2026-10-30 02:54:36.807313058 +0000 UTC m=+22.987040505"
Oct 30 02:54:37.348021 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:37.347990    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xg9mx"]
Oct 30 02:54:37.348318 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:37.348134    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:37.348318 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:37.348237    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:37.350779 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:37.350756    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ffnvh"]
Oct 30 02:54:37.350883 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:37.350845    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:37.350977 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:37.350960    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:37.351216 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:37.351197    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k6rks"]
Oct 30 02:54:37.351297 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:37.351286    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:37.351375 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:37.351361    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:38.405211 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:38.405180    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:38.405598 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:38.405282    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:39.405044 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:39.404803    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:39.405237 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:39.405208    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:39.405724 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:39.405700    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:39.405856 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:39.405833    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:40.404855 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:40.404825    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:40.405054 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:40.404956    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:41.404728 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:41.404689    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:41.404728 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:41.404714    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:41.405231 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:41.404810    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ffnvh" podUID="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee"
Oct 30 02:54:41.405231 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:41.404953    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k6rks" podUID="95a07519-99bf-47ac-b236-c642491d5648"
Oct 30 02:54:42.404826 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:42.404790    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:42.405239 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:42.404913    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xg9mx" podUID="23d4d339-696a-4b94-b34f-19852eb3cbc2"
Oct 30 02:54:43.068417 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.068348    2544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeReady"
Oct 30 02:54:43.068562 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.068467    2544 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 02:54:43.119465 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.119431    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zvmbf"]
Oct 30 02:54:43.136312 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.136275    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pkft5"]
Oct 30 02:54:43.136459 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.136399    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zvmbf"
Oct 30 02:54:43.146790 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.146757    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6qkm5"]
Oct 30 02:54:43.146930 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.146828    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pkft5"
Oct 30 02:54:43.149494 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.149472    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"]
Oct 30 02:54:43.149645 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.149577    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6qkm5"
Oct 30 02:54:43.152404 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.152386    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7zx8h"]
Oct 30 02:54:43.152523 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.152498    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:43.157548 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.157522    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2"]
Oct 30 02:54:43.157702 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.157664    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.158308 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.158289    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 02:54:43.160529 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.160507    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"]
Oct 30 02:54:43.160709 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.160691    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2"
Oct 30 02:54:43.165645 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.165600    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-57bd894fd7-n6trx"]
Oct 30 02:54:43.165950 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.165714    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:43.168449 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.168425    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 02:54:43.169196 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.169175    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qggb8"]
Oct 30 02:54:43.169344 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.169327    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.172769 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.172746    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"]
Oct 30 02:54:43.172923 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.172886    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.175694 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.175675    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bbffcc788-vdsgf"]
Oct 30 02:54:43.175887 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.175874    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:43.178817 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.178797    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"]
Oct 30 02:54:43.178948 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.178933    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.185480 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.185462    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"]
Oct 30 02:54:43.185624 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.185593    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.200715 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.200691    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"]
Oct 30 02:54:43.200886 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.200866    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:43.209493 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.209457    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"]
Oct 30 02:54:43.209703 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.209580    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:43.215331 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.215055    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2hw2r"]
Oct 30 02:54:43.216480 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.216459    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.220469 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.217980    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 02:54:43.220469 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.218009    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 02:54:43.220469 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.218070    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 02:54:43.220469 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.218110    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 02:54:43.220469 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.218151    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 02:54:43.220469 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.218174    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:54:43.220469 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.218188    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvbgx\""
Oct 30 02:54:43.220469 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.218346    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-hx2qx\""
Oct 30 02:54:43.220469 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.218651    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 02:54:43.234727 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.234296    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7"]
Oct 30 02:54:43.234818 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.234742    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2hw2r"
Oct 30 02:54:43.246599 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.246568    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pkft5"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.246666    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zvmbf"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.246715    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7"
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.246724    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6qkm5"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.246751    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7zx8h"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.246792    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.246814    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.246905    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.246929    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.246953    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-57bd894fd7-n6trx"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.246976    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.246999    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qggb8"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.247027    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.247056    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2hw2r"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.247078    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7"]
Oct 30 02:54:43.247093 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.247103    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"]
Oct 30 02:54:43.248203 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.248181    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"]
Oct 30 02:54:43.251139 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.251118    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bbffcc788-vdsgf"]
Oct 30 02:54:43.257724 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.257702    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 02:54:43.257836 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.257790    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:43.258033 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.258015    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:43.258145 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.258020    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:43.258145 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.258122    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:43.258288 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.258275    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 02:54:43.258288 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.258244    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 02:54:43.258701 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.258685    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 02:54:43.258787 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.258685    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:43.258787 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.258721    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:54:43.258787 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.258752    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 02:54:43.258907 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.258788    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:54:43.264629 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.264598    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:54:43.265556 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.265537    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ded308e2-be12-49e1-9ddc-153bedb23ce1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:43.265655 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.265562    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdsmr\" (UniqueName: \"kubernetes.io/projected/ded308e2-be12-49e1-9ddc-153bedb23ce1-kube-api-access-jdsmr\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:43.265655 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.265581    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed07eb99-f3f5-4dd2-8989-970f992de3fc-ca-trust-extracted\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.265655 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.265597    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-bound-sa-token\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.265794 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.265719    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:43.265794 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.265757    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sffkm\" (UniqueName: \"kubernetes.io/projected/dbb2d265-64b9-4e4f-be5b-1f00ba684673-kube-api-access-sffkm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:43.265892 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.265803    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d921d44f-5ee7-449a-87da-2ebbb52af1aa-config-volume\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.265892 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.265841    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-serving-cert\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.265892 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.265860    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e4c1ab5-989e-4fe6-a028-85579a18a996-tmp\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.265892 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.265878    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-certificates\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.266095 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.265938    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntrv\" (UniqueName: \"kubernetes.io/projected/def369e4-1dd4-49b8-8ee7-b6b21168a589-kube-api-access-qntrv\") pod \"kas-connection-checker-685895c7c9-zvmbf\" (UID: \"def369e4-1dd4-49b8-8ee7-b6b21168a589\") " pod="kube-system/kas-connection-checker-685895c7c9-zvmbf"
Oct 30 02:54:43.266095 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.265973    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d921d44f-5ee7-449a-87da-2ebbb52af1aa-tmp-dir\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.266095 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266001    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q85mq\" (UniqueName: \"kubernetes.io/projected/d921d44f-5ee7-449a-87da-2ebbb52af1aa-kube-api-access-q85mq\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.266095 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266029    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-trusted-ca\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.266095 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266074    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.266401 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266107    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-trusted-ca\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.266401 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266157    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.266401 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266216    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbwxs\" (UniqueName: \"kubernetes.io/projected/6f24e8ae-7115-48d9-b285-738049028e7b-kube-api-access-pbwxs\") pod \"kas-connection-checker-685895c7c9-pkft5\" (UID: \"6f24e8ae-7115-48d9-b285-738049028e7b\") " pod="kube-system/kas-connection-checker-685895c7c9-pkft5"
Oct 30 02:54:43.266401 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266242    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-bound-sa-token\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.266401 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266276    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-image-registry-private-configuration\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.266401 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266317    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.266401 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266356    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd3e5117-fce3-467f-946b-745eaa13f3e7-ca-trust-extracted\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.266899 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266404    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8tp\" (UniqueName: \"kubernetes.io/projected/2e4c1ab5-989e-4fe6-a028-85579a18a996-kube-api-access-9t8tp\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.266899 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266478    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-certificates\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.266899 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266508    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-trusted-ca\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.266899 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266541    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.266899 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266638    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c59c81e5-84cf-4660-9dd3-f3d8999812ec-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:43.266899 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266696    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edcb5539-95e1-4797-9a94-0f14ffe5ca00-serving-cert\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:43.266899 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266728    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4c1ab5-989e-4fe6-a028-85579a18a996-serving-cert\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.266899 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266764    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-installation-pull-secrets\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.266899 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266815    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29fjt\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-kube-api-access-29fjt\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.266899 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266884    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbb2d265-64b9-4e4f-be5b-1f00ba684673-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266914    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edcb5539-95e1-4797-9a94-0f14ffe5ca00-config\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266939    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266970    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e4c1ab5-989e-4fe6-a028-85579a18a996-snapshots\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.266990    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-config\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267005    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsrrk\" (UniqueName: \"kubernetes.io/projected/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-kube-api-access-gsrrk\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267020    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-default-certificate\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267043    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267097    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p9fq\" (UniqueName: \"kubernetes.io/projected/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-kube-api-access-5p9fq\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267143    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267163    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-installation-pull-secrets\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267183    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8pf\" (UniqueName: \"kubernetes.io/projected/edcb5539-95e1-4797-9a94-0f14ffe5ca00-kube-api-access-zq8pf\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267225    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-stats-auth\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.267256 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267257    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b2xf\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-kube-api-access-7b2xf\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.267664 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267280    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chclf\" (UniqueName: \"kubernetes.io/projected/4bfba258-be89-4d08-8691-1e7ccd648533-kube-api-access-chclf\") pod \"kas-connection-checker-685895c7c9-6qkm5\" (UID: \"4bfba258-be89-4d08-8691-1e7ccd648533\") " pod="kube-system/kas-connection-checker-685895c7c9-6qkm5"
Oct 30 02:54:43.267664 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267303    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4c1ab5-989e-4fe6-a028-85579a18a996-service-ca-bundle\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.267664 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267353    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7q55\" (UniqueName: \"kubernetes.io/projected/a09a8421-0c4e-4343-90ac-a3d3b7f9ba36-kube-api-access-m7q55\") pod \"volume-data-source-validator-58c9f5f46d-7fbf2\" (UID: \"a09a8421-0c4e-4343-90ac-a3d3b7f9ba36\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2"
Oct 30 02:54:43.267664 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267369    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbb2d265-64b9-4e4f-be5b-1f00ba684673-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:43.267664 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267386    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:43.267664 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267403    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4hqw\" (UniqueName: \"kubernetes.io/projected/768098d2-ba3f-41b8-8263-b74826c98c73-kube-api-access-g4hqw\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.267664 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267428    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-image-registry-private-configuration\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.267664 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.267457    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4c1ab5-989e-4fe6-a028-85579a18a996-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.271726 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.271709    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:43.272742 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.272727    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:43.272742 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.272734    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:54:43.272973 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.272940    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:43.273051 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.272977    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:54:43.279302 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.279284    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 02:54:43.279384 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.279357    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 02:54:43.279434 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.279420    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 02:54:43.286483 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.286462    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:43.286483 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.286464    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:54:43.287349 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.287337    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:54:43.287387 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.287352    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:54:43.287522 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.287508    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:43.302296 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.302262    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 02:54:43.316994 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.316967    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 02:54:43.331398 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.331328    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 02:54:43.331796 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.331779    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:54:43.331865 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.331825    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:54:43.332170 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.332154    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:54:43.332254 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.332172    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 02:54:43.335170 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.335154    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 02:54:43.335340 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.335323    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qzv89\""
Oct 30 02:54:43.335408 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.335364    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 02:54:43.335408 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.335386    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 02:54:43.335408 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.335399    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4tlgk\""
Oct 30 02:54:43.335508 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.335413    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 02:54:43.338326 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.338298    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 02:54:43.338326 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.338304    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j9c5h\""
Oct 30 02:54:43.338326 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.338310    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:54:43.338537 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.338354    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 02:54:43.368409 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.368375    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hqw\" (UniqueName: \"kubernetes.io/projected/768098d2-ba3f-41b8-8263-b74826c98c73-kube-api-access-g4hqw\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.368409 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.368414    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0-cert\") pod \"ingress-canary-2hw2r\" (UID: \"5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0\") " pod="openshift-ingress-canary/ingress-canary-2hw2r"
Oct 30 02:54:43.368579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.368437    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-image-registry-private-configuration\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.368579 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.368481    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4c1ab5-989e-4fe6-a028-85579a18a996-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.368711 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.368670    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ded308e2-be12-49e1-9ddc-153bedb23ce1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:43.368760 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.368710    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdsmr\" (UniqueName: \"kubernetes.io/projected/ded308e2-be12-49e1-9ddc-153bedb23ce1-kube-api-access-jdsmr\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:43.368852 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.368829    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed07eb99-f3f5-4dd2-8989-970f992de3fc-ca-trust-extracted\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.368911 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.368867    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-bound-sa-token\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.368984 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.368936    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:43.369076 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.368981    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffkm\" (UniqueName: \"kubernetes.io/projected/dbb2d265-64b9-4e4f-be5b-1f00ba684673-kube-api-access-sffkm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:43.369076 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369009    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d921d44f-5ee7-449a-87da-2ebbb52af1aa-config-volume\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.369076 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369032    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-serving-cert\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.369076 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369068    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e4c1ab5-989e-4fe6-a028-85579a18a996-tmp\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.369245 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369098    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-certificates\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.369245 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369127    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntrv\" (UniqueName: \"kubernetes.io/projected/def369e4-1dd4-49b8-8ee7-b6b21168a589-kube-api-access-qntrv\") pod \"kas-connection-checker-685895c7c9-zvmbf\" (UID: \"def369e4-1dd4-49b8-8ee7-b6b21168a589\") " pod="kube-system/kas-connection-checker-685895c7c9-zvmbf"
Oct 30 02:54:43.369245 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369156    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d921d44f-5ee7-449a-87da-2ebbb52af1aa-tmp-dir\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.369245 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369181    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85mq\" (UniqueName: \"kubernetes.io/projected/d921d44f-5ee7-449a-87da-2ebbb52af1aa-kube-api-access-q85mq\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.369245 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369206    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-trusted-ca\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.369419 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369237    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed07eb99-f3f5-4dd2-8989-970f992de3fc-ca-trust-extracted\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.369419 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369248    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.369419 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369273    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-trusted-ca\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.369419 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369297    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.369419 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369318    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwxs\" (UniqueName: \"kubernetes.io/projected/6f24e8ae-7115-48d9-b285-738049028e7b-kube-api-access-pbwxs\") pod \"kas-connection-checker-685895c7c9-pkft5\" (UID: \"6f24e8ae-7115-48d9-b285-738049028e7b\") " pod="kube-system/kas-connection-checker-685895c7c9-pkft5"
Oct 30 02:54:43.369419 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369344    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-bound-sa-token\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.369419 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.369356    2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:54:43.369662 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.369433    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls podName:ded308e2-be12-49e1-9ddc-153bedb23ce1 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:43.869406736 +0000 UTC m=+30.049134161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c2fsj" (UID: "ded308e2-be12-49e1-9ddc-153bedb23ce1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:54:43.369662 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369434    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ded308e2-be12-49e1-9ddc-153bedb23ce1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:43.369662 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.369461    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:43.369662 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.369476    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cc9c5bc45-q8v5t: secret "image-registry-tls" not found
Oct 30 02:54:43.369662 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.369532    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls podName:ed07eb99-f3f5-4dd2-8989-970f992de3fc nodeName:}" failed. No retries permitted until 2026-10-30 02:54:43.869517194 +0000 UTC m=+30.049244618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls") pod "image-registry-7cc9c5bc45-q8v5t" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc") : secret "image-registry-tls" not found
Oct 30 02:54:43.369906 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369719    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e4c1ab5-989e-4fe6-a028-85579a18a996-tmp\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.369906 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.369722    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:43.369906 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.369777    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls podName:d921d44f-5ee7-449a-87da-2ebbb52af1aa nodeName:}" failed. No retries permitted until 2026-10-30 02:54:43.869763792 +0000 UTC m=+30.049491217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls") pod "dns-default-qggb8" (UID: "d921d44f-5ee7-449a-87da-2ebbb52af1aa") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:43.369906 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369791    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4c1ab5-989e-4fe6-a028-85579a18a996-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.369906 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.369887    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d921d44f-5ee7-449a-87da-2ebbb52af1aa-tmp-dir\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.370152 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370009    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6t8d\" (UniqueName: \"kubernetes.io/projected/083a96ad-9c35-4895-be37-3b8de8089360-kube-api-access-w6t8d\") pod \"network-check-source-6f78d59f99-zq8h7\" (UID: \"083a96ad-9c35-4895-be37-3b8de8089360\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7"
Oct 30 02:54:43.370152 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370044    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-image-registry-private-configuration\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.370152 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370068    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.370152 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370053    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-certificates\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.370152 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370119    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q95nw\" (UniqueName: \"kubernetes.io/projected/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0-kube-api-access-q95nw\") pod \"ingress-canary-2hw2r\" (UID: \"5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0\") " pod="openshift-ingress-canary/ingress-canary-2hw2r"
Oct 30 02:54:43.370373 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370181    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd3e5117-fce3-467f-946b-745eaa13f3e7-ca-trust-extracted\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.370373 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.370198    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:43.370373 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.370212    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bbffcc788-vdsgf: secret "image-registry-tls" not found
Oct 30 02:54:43.370373 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370215    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8tp\" (UniqueName: \"kubernetes.io/projected/2e4c1ab5-989e-4fe6-a028-85579a18a996-kube-api-access-9t8tp\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.370373 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.370255    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls podName:bd3e5117-fce3-467f-946b-745eaa13f3e7 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:43.870240362 +0000 UTC m=+30.049967800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls") pod "image-registry-5bbffcc788-vdsgf" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7") : secret "image-registry-tls" not found
Oct 30 02:54:43.370373 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370276    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-certificates\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.370373 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370308    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-trusted-ca\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.370373 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370336    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.370373 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370368    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c59c81e5-84cf-4660-9dd3-f3d8999812ec-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:43.370814 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370397    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edcb5539-95e1-4797-9a94-0f14ffe5ca00-serving-cert\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:43.370814 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370422    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4c1ab5-989e-4fe6-a028-85579a18a996-serving-cert\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.370814 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370428    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d921d44f-5ee7-449a-87da-2ebbb52af1aa-config-volume\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.370814 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370449    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-installation-pull-secrets\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.370814 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370476    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29fjt\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-kube-api-access-29fjt\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.370814 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370763    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-trusted-ca\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.371096 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.370976    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:43.870958974 +0000 UTC m=+30.050686422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : configmap references non-existent config key: service-ca.crt
Oct 30 02:54:43.371096 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.370984    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd3e5117-fce3-467f-946b-745eaa13f3e7-ca-trust-extracted\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.371262 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.371226    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c59c81e5-84cf-4660-9dd3-f3d8999812ec-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:43.371352 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.371273    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-trusted-ca\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.371352 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.371330    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-trusted-ca\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.371506 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.371363    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbb2d265-64b9-4e4f-be5b-1f00ba684673-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:43.371677 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.371581    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-certificates\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.371893 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.371765    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edcb5539-95e1-4797-9a94-0f14ffe5ca00-config\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:43.372192 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.372156    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:43.372338 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.372322    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e4c1ab5-989e-4fe6-a028-85579a18a996-snapshots\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.372460 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.372445    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-config\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.372583 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.372568    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsrrk\" (UniqueName: \"kubernetes.io/projected/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-kube-api-access-gsrrk\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.372729 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.372711    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-default-certificate\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.372867 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.372843    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e4c1ab5-989e-4fe6-a028-85579a18a996-snapshots\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.372947 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.372847    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:43.372947 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.372915    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p9fq\" (UniqueName: \"kubernetes.io/projected/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-kube-api-access-5p9fq\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:43.373054 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.372945    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.373054 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.372973    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-installation-pull-secrets\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.373054 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.372997    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8pf\" (UniqueName: \"kubernetes.io/projected/edcb5539-95e1-4797-9a94-0f14ffe5ca00-kube-api-access-zq8pf\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:43.373054 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.373022    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-stats-auth\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.373054 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.373048    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b2xf\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-kube-api-access-7b2xf\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.373283 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.373069    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chclf\" (UniqueName: \"kubernetes.io/projected/4bfba258-be89-4d08-8691-1e7ccd648533-kube-api-access-chclf\") pod \"kas-connection-checker-685895c7c9-6qkm5\" (UID: \"4bfba258-be89-4d08-8691-1e7ccd648533\") " pod="kube-system/kas-connection-checker-685895c7c9-6qkm5"
Oct 30 02:54:43.373283 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.373090    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4c1ab5-989e-4fe6-a028-85579a18a996-service-ca-bundle\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.373283 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.373137    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7q55\" (UniqueName: \"kubernetes.io/projected/a09a8421-0c4e-4343-90ac-a3d3b7f9ba36-kube-api-access-m7q55\") pod \"volume-data-source-validator-58c9f5f46d-7fbf2\" (UID: \"a09a8421-0c4e-4343-90ac-a3d3b7f9ba36\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2"
Oct 30 02:54:43.373283 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.373160    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbb2d265-64b9-4e4f-be5b-1f00ba684673-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:43.373283 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.373184    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:43.373531 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.372258    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edcb5539-95e1-4797-9a94-0f14ffe5ca00-config\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:43.373707 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.372345    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:54:43.373707 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.373641    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert podName:c59c81e5-84cf-4660-9dd3-f3d8999812ec nodeName:}" failed. No retries permitted until 2026-10-30 02:54:43.873597909 +0000 UTC m=+30.053325348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5wtlc" (UID: "c59c81e5-84cf-4660-9dd3-f3d8999812ec") : secret "networking-console-plugin-cert" not found
Oct 30 02:54:43.373969 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.373868    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-serving-cert\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.373969 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.373925    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-image-registry-private-configuration\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.374332 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.374263    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbb2d265-64b9-4e4f-be5b-1f00ba684673-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:43.374332 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.374265    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-config\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.374332 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.374298    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-installation-pull-secrets\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.374332 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.373312    2544 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:54:43.374550 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.374364    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls podName:ea4de3ea-99e3-4984-bae1-5f3ae9e4460a nodeName:}" failed. No retries permitted until 2026-10-30 02:54:43.874350206 +0000 UTC m=+30.054077632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fx5sl" (UID: "ea4de3ea-99e3-4984-bae1-5f3ae9e4460a") : secret "samples-operator-tls" not found
Oct 30 02:54:43.374550 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.373538    2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:54:43.374550 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.374404    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:43.874393787 +0000 UTC m=+30.054121237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : secret "router-metrics-certs-default" not found
Oct 30 02:54:43.374897 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.374874    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:43.375269 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.375076    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4c1ab5-989e-4fe6-a028-85579a18a996-service-ca-bundle\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.375352 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.375282    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edcb5539-95e1-4797-9a94-0f14ffe5ca00-serving-cert\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:43.375516 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.375496    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4c1ab5-989e-4fe6-a028-85579a18a996-serving-cert\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.375914 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.375692    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbb2d265-64b9-4e4f-be5b-1f00ba684673-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:43.389079 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.389032    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-stats-auth\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.389079 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.389038    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-default-certificate\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.389263 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.389109    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-installation-pull-secrets\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.389263 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.389151    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-image-registry-private-configuration\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.395221 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.395187    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hqw\" (UniqueName: \"kubernetes.io/projected/768098d2-ba3f-41b8-8263-b74826c98c73-kube-api-access-g4hqw\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.395648 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.395381    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffkm\" (UniqueName: \"kubernetes.io/projected/dbb2d265-64b9-4e4f-be5b-1f00ba684673-kube-api-access-sffkm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:43.395648 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.395415    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdsmr\" (UniqueName: \"kubernetes.io/projected/ded308e2-be12-49e1-9ddc-153bedb23ce1-kube-api-access-jdsmr\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:43.395648 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.395426    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntrv\" (UniqueName: \"kubernetes.io/projected/def369e4-1dd4-49b8-8ee7-b6b21168a589-kube-api-access-qntrv\") pod \"kas-connection-checker-685895c7c9-zvmbf\" (UID: \"def369e4-1dd4-49b8-8ee7-b6b21168a589\") " pod="kube-system/kas-connection-checker-685895c7c9-zvmbf"
Oct 30 02:54:43.396184 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.396152    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbwxs\" (UniqueName: \"kubernetes.io/projected/6f24e8ae-7115-48d9-b285-738049028e7b-kube-api-access-pbwxs\") pod \"kas-connection-checker-685895c7c9-pkft5\" (UID: \"6f24e8ae-7115-48d9-b285-738049028e7b\") " pod="kube-system/kas-connection-checker-685895c7c9-pkft5"
Oct 30 02:54:43.396632 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.396587    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85mq\" (UniqueName: \"kubernetes.io/projected/d921d44f-5ee7-449a-87da-2ebbb52af1aa-kube-api-access-q85mq\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.397534 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.397507    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-bound-sa-token\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.398222 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.398197    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8tp\" (UniqueName: \"kubernetes.io/projected/2e4c1ab5-989e-4fe6-a028-85579a18a996-kube-api-access-9t8tp\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.398332 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.398310    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29fjt\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-kube-api-access-29fjt\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.398846 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.398824    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p9fq\" (UniqueName: \"kubernetes.io/projected/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-kube-api-access-5p9fq\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:43.399835 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.399811    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsrrk\" (UniqueName: \"kubernetes.io/projected/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-kube-api-access-gsrrk\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.400285 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.400264    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7q55\" (UniqueName: \"kubernetes.io/projected/a09a8421-0c4e-4343-90ac-a3d3b7f9ba36-kube-api-access-m7q55\") pod \"volume-data-source-validator-58c9f5f46d-7fbf2\" (UID: \"a09a8421-0c4e-4343-90ac-a3d3b7f9ba36\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2"
Oct 30 02:54:43.401073 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.401054    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chclf\" (UniqueName: \"kubernetes.io/projected/4bfba258-be89-4d08-8691-1e7ccd648533-kube-api-access-chclf\") pod \"kas-connection-checker-685895c7c9-6qkm5\" (UID: \"4bfba258-be89-4d08-8691-1e7ccd648533\") " pod="kube-system/kas-connection-checker-685895c7c9-6qkm5"
Oct 30 02:54:43.402254 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.402230    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8pf\" (UniqueName: \"kubernetes.io/projected/edcb5539-95e1-4797-9a94-0f14ffe5ca00-kube-api-access-zq8pf\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:43.402532 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.402510    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b2xf\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-kube-api-access-7b2xf\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.404683 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.404664    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:43.404768 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.404609    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:43.404860 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.404842    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-bound-sa-token\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.424027 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.424000    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l729x\""
Oct 30 02:54:43.424167 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.424057    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:54:43.424167 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.424084    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d6p9p\""
Oct 30 02:54:43.424167 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.424094    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:54:43.432176 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.432153    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:43.443711 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.443684    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:43.448123 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.448099    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zvmbf"
Oct 30 02:54:43.460176 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.460148    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pkft5"
Oct 30 02:54:43.473831 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.473801    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6qkm5"
Oct 30 02:54:43.474361 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.474318    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6t8d\" (UniqueName: \"kubernetes.io/projected/083a96ad-9c35-4895-be37-3b8de8089360-kube-api-access-w6t8d\") pod \"network-check-source-6f78d59f99-zq8h7\" (UID: \"083a96ad-9c35-4895-be37-3b8de8089360\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7"
Oct 30 02:54:43.474466 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.474371    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q95nw\" (UniqueName: \"kubernetes.io/projected/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0-kube-api-access-q95nw\") pod \"ingress-canary-2hw2r\" (UID: \"5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0\") " pod="openshift-ingress-canary/ingress-canary-2hw2r"
Oct 30 02:54:43.474576 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.474558    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0-cert\") pod \"ingress-canary-2hw2r\" (UID: \"5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0\") " pod="openshift-ingress-canary/ingress-canary-2hw2r"
Oct 30 02:54:43.478759 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.478729    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0-cert\") pod \"ingress-canary-2hw2r\" (UID: \"5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0\") " pod="openshift-ingress-canary/ingress-canary-2hw2r"
Oct 30 02:54:43.508389 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.505533    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6t8d\" (UniqueName: \"kubernetes.io/projected/083a96ad-9c35-4895-be37-3b8de8089360-kube-api-access-w6t8d\") pod \"network-check-source-6f78d59f99-zq8h7\" (UID: \"083a96ad-9c35-4895-be37-3b8de8089360\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7"
Oct 30 02:54:43.508389 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.508156    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q95nw\" (UniqueName: \"kubernetes.io/projected/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0-kube-api-access-q95nw\") pod \"ingress-canary-2hw2r\" (UID: \"5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0\") " pod="openshift-ingress-canary/ingress-canary-2hw2r"
Oct 30 02:54:43.509561 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.509048    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:43.514557 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.514037    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2"
Oct 30 02:54:43.555511 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.554864    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:43.715546 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.715515    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2hw2r"
Oct 30 02:54:43.754602 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.754560    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7"
Oct 30 02:54:43.782354 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.782330    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"]
Oct 30 02:54:43.786408 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:43.786376    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb2d265_64b9_4e4f_be5b_1f00ba684673.slice/crio-97f0bce6196b817ebf333efbaa33d6e3a93c4f882316b4c029ad038169470d04 WatchSource:0}: Error finding container 97f0bce6196b817ebf333efbaa33d6e3a93c4f882316b4c029ad038169470d04: Status 404 returned error can't find the container with id 97f0bce6196b817ebf333efbaa33d6e3a93c4f882316b4c029ad038169470d04
Oct 30 02:54:43.788741 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.788256    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"]
Oct 30 02:54:43.793081 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.793055    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zvmbf"]
Oct 30 02:54:43.796346 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:43.796317    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b4da6f_8901_46e0_a9c8_fe3d1bea91a7.slice/crio-753770bea8207aebe5b2731bbd5cc3d726692083dedff93f492645a56a14ec07 WatchSource:0}: Error finding container 753770bea8207aebe5b2731bbd5cc3d726692083dedff93f492645a56a14ec07: Status 404 returned error can't find the container with id 753770bea8207aebe5b2731bbd5cc3d726692083dedff93f492645a56a14ec07
Oct 30 02:54:43.797062 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:43.796608    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef369e4_1dd4_49b8_8ee7_b6b21168a589.slice/crio-d5b3b707f2d875ae9fa9d6397f686fe0b18b0ef0005e7be19ff70beb659b50bc WatchSource:0}: Error finding container d5b3b707f2d875ae9fa9d6397f686fe0b18b0ef0005e7be19ff70beb659b50bc: Status 404 returned error can't find the container with id d5b3b707f2d875ae9fa9d6397f686fe0b18b0ef0005e7be19ff70beb659b50bc
Oct 30 02:54:43.838817 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.837978    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6qkm5"]
Oct 30 02:54:43.842604 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:43.842569    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfba258_be89_4d08_8691_1e7ccd648533.slice/crio-08406e1cdab3a28fc3a51f0c5c3a03cd3669f17c9ef57373dc314d9a36bc258c WatchSource:0}: Error finding container 08406e1cdab3a28fc3a51f0c5c3a03cd3669f17c9ef57373dc314d9a36bc258c: Status 404 returned error can't find the container with id 08406e1cdab3a28fc3a51f0c5c3a03cd3669f17c9ef57373dc314d9a36bc258c
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.883028    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.883093    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.883122    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.883153    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.883187    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.883231    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.883263    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.883309    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.883445    2544 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.883502    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls podName:ea4de3ea-99e3-4984-bae1-5f3ae9e4460a nodeName:}" failed. No retries permitted until 2026-10-30 02:54:44.883483608 +0000 UTC m=+31.063211047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fx5sl" (UID: "ea4de3ea-99e3-4984-bae1-5f3ae9e4460a") : secret "samples-operator-tls" not found
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.883520    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7zx8h"]
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.883561    2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.883594    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls podName:ded308e2-be12-49e1-9ddc-153bedb23ce1 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:44.8835835 +0000 UTC m=+31.063310937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c2fsj" (UID: "ded308e2-be12-49e1-9ddc-153bedb23ce1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.883766    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:44.883751871 +0000 UTC m=+31.063479309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : configmap references non-existent config key: service-ca.crt
Oct 30 02:54:43.884198 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.883829    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:43.885333 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.883860    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls podName:d921d44f-5ee7-449a-87da-2ebbb52af1aa nodeName:}" failed. No retries permitted until 2026-10-30 02:54:44.883849037 +0000 UTC m=+31.063576475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls") pod "dns-default-qggb8" (UID: "d921d44f-5ee7-449a-87da-2ebbb52af1aa") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:43.885333 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.883915    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:43.885333 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.883926    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bbffcc788-vdsgf: secret "image-registry-tls" not found
Oct 30 02:54:43.885333 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.883954    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls podName:bd3e5117-fce3-467f-946b-745eaa13f3e7 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:44.883944629 +0000 UTC m=+31.063672069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls") pod "image-registry-5bbffcc788-vdsgf" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7") : secret "image-registry-tls" not found
Oct 30 02:54:43.885333 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.884007    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:54:43.885333 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.884036    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert podName:c59c81e5-84cf-4660-9dd3-f3d8999812ec nodeName:}" failed. No retries permitted until 2026-10-30 02:54:44.884026136 +0000 UTC m=+31.063753578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5wtlc" (UID: "c59c81e5-84cf-4660-9dd3-f3d8999812ec") : secret "networking-console-plugin-cert" not found
Oct 30 02:54:43.885333 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.884117    2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:54:43.885333 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.884145    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:44.884136258 +0000 UTC m=+31.063863698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : secret "router-metrics-certs-default" not found
Oct 30 02:54:43.885333 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.884449    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:43.885333 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.884502    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cc9c5bc45-q8v5t: secret "image-registry-tls" not found
Oct 30 02:54:43.885333 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:43.884542    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls podName:ed07eb99-f3f5-4dd2-8989-970f992de3fc nodeName:}" failed. No retries permitted until 2026-10-30 02:54:44.88452795 +0000 UTC m=+31.064255393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls") pod "image-registry-7cc9c5bc45-q8v5t" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc") : secret "image-registry-tls" not found
Oct 30 02:54:43.888368 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.887432    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pkft5"]
Oct 30 02:54:43.891906 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:43.889425    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f24e8ae_7115_48d9_b285_738049028e7b.slice/crio-e0a45025b973a7465a6f421feb787efcdb9003fb31bdb25009faeb9b5459c19c WatchSource:0}: Error finding container e0a45025b973a7465a6f421feb787efcdb9003fb31bdb25009faeb9b5459c19c: Status 404 returned error can't find the container with id e0a45025b973a7465a6f421feb787efcdb9003fb31bdb25009faeb9b5459c19c
Oct 30 02:54:43.891906 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:43.890801    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4c1ab5_989e_4fe6_a028_85579a18a996.slice/crio-b41bfb0e51691b48fb41455733a8ab7fbb8eeff910734c1496d3ecaaa534bba0 WatchSource:0}: Error finding container b41bfb0e51691b48fb41455733a8ab7fbb8eeff910734c1496d3ecaaa534bba0: Status 404 returned error can't find the container with id b41bfb0e51691b48fb41455733a8ab7fbb8eeff910734c1496d3ecaaa534bba0
Oct 30 02:54:43.907166 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.901341    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2"]
Oct 30 02:54:43.907166 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:43.904468    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09a8421_0c4e_4343_90ac_a3d3b7f9ba36.slice/crio-315e6fc062942a3bdec1f4bc8bd87db05b7b437cabe827b9d5af393b3f336043 WatchSource:0}: Error finding container 315e6fc062942a3bdec1f4bc8bd87db05b7b437cabe827b9d5af393b3f336043: Status 404 returned error can't find the container with id 315e6fc062942a3bdec1f4bc8bd87db05b7b437cabe827b9d5af393b3f336043
Oct 30 02:54:43.924152 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:43.924125    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"]
Oct 30 02:54:43.932908 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:43.932877    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcb5539_95e1_4797_9a94_0f14ffe5ca00.slice/crio-5d1fb988c7d2fe5cd5fa417c7bdcb8f549738ff6c69864b7331919e5bb8ae5d3 WatchSource:0}: Error finding container 5d1fb988c7d2fe5cd5fa417c7bdcb8f549738ff6c69864b7331919e5bb8ae5d3: Status 404 returned error can't find the container with id 5d1fb988c7d2fe5cd5fa417c7bdcb8f549738ff6c69864b7331919e5bb8ae5d3
Oct 30 02:54:44.028538 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.028511    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2hw2r"]
Oct 30 02:54:44.028690 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:44.028670    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c66fc5b_5ff8_4e2b_9192_4c6b38e5e1c0.slice/crio-9d4692a6677681ecf4af55e75fb85429db38e9eec8ebb9bec1d9f7c466276209 WatchSource:0}: Error finding container 9d4692a6677681ecf4af55e75fb85429db38e9eec8ebb9bec1d9f7c466276209: Status 404 returned error can't find the container with id 9d4692a6677681ecf4af55e75fb85429db38e9eec8ebb9bec1d9f7c466276209
Oct 30 02:54:44.099516 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.099289    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7"]
Oct 30 02:54:44.100339 ip-10-0-140-132 kubenswrapper[2544]: W1030 02:54:44.100316    2544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083a96ad_9c35_4895_be37_3b8de8089360.slice/crio-27e8b88a4c04f0515edd8aef914df11f5ecd87460697ba48feda0f14e059b734 WatchSource:0}: Error finding container 27e8b88a4c04f0515edd8aef914df11f5ecd87460697ba48feda0f14e059b734: Status 404 returned error can't find the container with id 27e8b88a4c04f0515edd8aef914df11f5ecd87460697ba48feda0f14e059b734
Oct 30 02:54:44.407003 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.406928    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:44.425602 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.425572    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-h9jxb\""
Oct 30 02:54:44.722607 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.722469    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2hw2r" event={"ID":"5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0","Type":"ContainerStarted","Data":"9d4692a6677681ecf4af55e75fb85429db38e9eec8ebb9bec1d9f7c466276209"}
Oct 30 02:54:44.726400 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.726306    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2" event={"ID":"a09a8421-0c4e-4343-90ac-a3d3b7f9ba36","Type":"ContainerStarted","Data":"315e6fc062942a3bdec1f4bc8bd87db05b7b437cabe827b9d5af393b3f336043"}
Oct 30 02:54:44.728257 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.728175    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7zx8h" event={"ID":"2e4c1ab5-989e-4fe6-a028-85579a18a996","Type":"ContainerStarted","Data":"b41bfb0e51691b48fb41455733a8ab7fbb8eeff910734c1496d3ecaaa534bba0"}
Oct 30 02:54:44.730032 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.729957    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt" event={"ID":"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7","Type":"ContainerStarted","Data":"753770bea8207aebe5b2731bbd5cc3d726692083dedff93f492645a56a14ec07"}
Oct 30 02:54:44.732511 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.731718    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pkft5" event={"ID":"6f24e8ae-7115-48d9-b285-738049028e7b","Type":"ContainerStarted","Data":"1b8821cb6b5f455b6bcff84560a98ac274931363562a1106c27ba066b0e06afc"}
Oct 30 02:54:44.732511 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.731770    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pkft5" event={"ID":"6f24e8ae-7115-48d9-b285-738049028e7b","Type":"ContainerStarted","Data":"e0a45025b973a7465a6f421feb787efcdb9003fb31bdb25009faeb9b5459c19c"}
Oct 30 02:54:44.734380 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.734292    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7" event={"ID":"083a96ad-9c35-4895-be37-3b8de8089360","Type":"ContainerStarted","Data":"27e8b88a4c04f0515edd8aef914df11f5ecd87460697ba48feda0f14e059b734"}
Oct 30 02:54:44.736955 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.736877    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5" event={"ID":"edcb5539-95e1-4797-9a94-0f14ffe5ca00","Type":"ContainerStarted","Data":"5d1fb988c7d2fe5cd5fa417c7bdcb8f549738ff6c69864b7331919e5bb8ae5d3"}
Oct 30 02:54:44.744699 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.744083    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zvmbf" event={"ID":"def369e4-1dd4-49b8-8ee7-b6b21168a589","Type":"ContainerStarted","Data":"6a7a5de1d16dba6a2d64ecfe12e9f9dea6b420cd761a7b30dd8524f25c7a0b5e"}
Oct 30 02:54:44.744699 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.744136    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zvmbf" event={"ID":"def369e4-1dd4-49b8-8ee7-b6b21168a589","Type":"ContainerStarted","Data":"d5b3b707f2d875ae9fa9d6397f686fe0b18b0ef0005e7be19ff70beb659b50bc"}
Oct 30 02:54:44.751583 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.751284    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6qkm5" event={"ID":"4bfba258-be89-4d08-8691-1e7ccd648533","Type":"ContainerStarted","Data":"601e6b587c4ff13500a5167d51602b11ddc08814a309d1fe87599d061e371600"}
Oct 30 02:54:44.751583 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.751343    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6qkm5" event={"ID":"4bfba258-be89-4d08-8691-1e7ccd648533","Type":"ContainerStarted","Data":"08406e1cdab3a28fc3a51f0c5c3a03cd3669f17c9ef57373dc314d9a36bc258c"}
Oct 30 02:54:44.757724 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.757640    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8" event={"ID":"dbb2d265-64b9-4e4f-be5b-1f00ba684673","Type":"ContainerStarted","Data":"97f0bce6196b817ebf333efbaa33d6e3a93c4f882316b4c029ad038169470d04"}
Oct 30 02:54:44.779568 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.778106    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pkft5" podStartSLOduration=379.778089138 podStartE2EDuration="6m19.778089138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:44.777815565 +0000 UTC m=+30.957543014" watchObservedRunningTime="2026-10-30 02:54:44.778089138 +0000 UTC m=+30.957816586"
Oct 30 02:54:44.825501 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.824466    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zvmbf" podStartSLOduration=379.824447652 podStartE2EDuration="6m19.824447652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:44.823695774 +0000 UTC m=+31.003423220" watchObservedRunningTime="2026-10-30 02:54:44.824447652 +0000 UTC m=+31.004175100"
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.894694    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.894794    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.894874    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.894907    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.894945    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.894983    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.895030    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:44.895065    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.895885    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.895941    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls podName:d921d44f-5ee7-449a-87da-2ebbb52af1aa nodeName:}" failed. No retries permitted until 2026-10-30 02:54:46.895927627 +0000 UTC m=+33.075655052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls") pod "dns-default-qggb8" (UID: "d921d44f-5ee7-449a-87da-2ebbb52af1aa") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.896261    2544 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.896297    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls podName:ea4de3ea-99e3-4984-bae1-5f3ae9e4460a nodeName:}" failed. No retries permitted until 2026-10-30 02:54:46.89628672 +0000 UTC m=+33.076014145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fx5sl" (UID: "ea4de3ea-99e3-4984-bae1-5f3ae9e4460a") : secret "samples-operator-tls" not found
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.896333    2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.896353    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls podName:ded308e2-be12-49e1-9ddc-153bedb23ce1 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:46.896347021 +0000 UTC m=+33.076074458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c2fsj" (UID: "ded308e2-be12-49e1-9ddc-153bedb23ce1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.896909    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:44.897338 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.896923    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cc9c5bc45-q8v5t: secret "image-registry-tls" not found
Oct 30 02:54:44.898215 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.896960    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls podName:ed07eb99-f3f5-4dd2-8989-970f992de3fc nodeName:}" failed. No retries permitted until 2026-10-30 02:54:46.896946721 +0000 UTC m=+33.076674160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls") pod "image-registry-7cc9c5bc45-q8v5t" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc") : secret "image-registry-tls" not found
Oct 30 02:54:44.898215 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.897037    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:46.897025052 +0000 UTC m=+33.076752495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : configmap references non-existent config key: service-ca.crt
Oct 30 02:54:44.898215 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.897098    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:44.898215 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.897109    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bbffcc788-vdsgf: secret "image-registry-tls" not found
Oct 30 02:54:44.898215 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.897139    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls podName:bd3e5117-fce3-467f-946b-745eaa13f3e7 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:46.897129494 +0000 UTC m=+33.076856934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls") pod "image-registry-5bbffcc788-vdsgf" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7") : secret "image-registry-tls" not found
Oct 30 02:54:44.898215 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.897199    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:54:44.898215 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.897229    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert podName:c59c81e5-84cf-4660-9dd3-f3d8999812ec nodeName:}" failed. No retries permitted until 2026-10-30 02:54:46.897219755 +0000 UTC m=+33.076947195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5wtlc" (UID: "c59c81e5-84cf-4660-9dd3-f3d8999812ec") : secret "networking-console-plugin-cert" not found
Oct 30 02:54:44.898215 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.897285    2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:54:44.898215 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:44.897316    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:46.89730658 +0000 UTC m=+33.077034019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : secret "router-metrics-certs-default" not found
Oct 30 02:54:46.770782 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:46.770744    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2" event={"ID":"a09a8421-0c4e-4343-90ac-a3d3b7f9ba36","Type":"ContainerStarted","Data":"e89fa5b2a3c68117731a553698dab691b2cd98d7c10165ea74aa11792b0db602"}
Oct 30 02:54:46.821065 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:46.821003    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6qkm5" podStartSLOduration=381.82098382 podStartE2EDuration="6m21.82098382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:44.87485792 +0000 UTC m=+31.054585391" watchObservedRunningTime="2026-10-30 02:54:46.82098382 +0000 UTC m=+33.000711269"
Oct 30 02:54:46.918744 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:46.918706    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:46.918744 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:46.918744    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:46.918960 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:46.918769    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:46.918960 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:46.918809    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:46.918960 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:46.918857    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:46.918960 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:46.918895    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:46.919152 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:46.918960    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:46.919152 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:46.919018    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:46.919226 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.919162    2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:54:46.919263 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.919225    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls podName:ded308e2-be12-49e1-9ddc-153bedb23ce1 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:50.919208127 +0000 UTC m=+37.098935570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c2fsj" (UID: "ded308e2-be12-49e1-9ddc-153bedb23ce1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:54:46.919673 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.919657    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:46.919749 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.919676    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cc9c5bc45-q8v5t: secret "image-registry-tls" not found
Oct 30 02:54:46.919749 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.919725    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls podName:ed07eb99-f3f5-4dd2-8989-970f992de3fc nodeName:}" failed. No retries permitted until 2026-10-30 02:54:50.919709933 +0000 UTC m=+37.099437364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls") pod "image-registry-7cc9c5bc45-q8v5t" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc") : secret "image-registry-tls" not found
Oct 30 02:54:46.919956 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.919936    2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:54:46.920015 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.919987    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:50.919972672 +0000 UTC m=+37.099700100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : secret "router-metrics-certs-default" not found
Oct 30 02:54:46.920063 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.920052    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:46.920096 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.920067    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bbffcc788-vdsgf: secret "image-registry-tls" not found
Oct 30 02:54:46.920127 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.920098    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls podName:bd3e5117-fce3-467f-946b-745eaa13f3e7 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:50.920087592 +0000 UTC m=+37.099815021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls") pod "image-registry-5bbffcc788-vdsgf" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7") : secret "image-registry-tls" not found
Oct 30 02:54:46.920173 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.920163    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:50.920152395 +0000 UTC m=+37.099879821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : configmap references non-existent config key: service-ca.crt
Oct 30 02:54:46.920231 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.920222    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:54:46.920261 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.920255    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert podName:c59c81e5-84cf-4660-9dd3-f3d8999812ec nodeName:}" failed. No retries permitted until 2026-10-30 02:54:50.920244832 +0000 UTC m=+37.099972263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5wtlc" (UID: "c59c81e5-84cf-4660-9dd3-f3d8999812ec") : secret "networking-console-plugin-cert" not found
Oct 30 02:54:46.920321 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.920306    2544 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:54:46.920365 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.920343    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls podName:ea4de3ea-99e3-4984-bae1-5f3ae9e4460a nodeName:}" failed. No retries permitted until 2026-10-30 02:54:50.920331201 +0000 UTC m=+37.100058628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fx5sl" (UID: "ea4de3ea-99e3-4984-bae1-5f3ae9e4460a") : secret "samples-operator-tls" not found
Oct 30 02:54:46.921340 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.919801    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:46.921340 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:46.921266    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls podName:d921d44f-5ee7-449a-87da-2ebbb52af1aa nodeName:}" failed. No retries permitted until 2026-10-30 02:54:50.921252609 +0000 UTC m=+37.100980035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls") pod "dns-default-qggb8" (UID: "d921d44f-5ee7-449a-87da-2ebbb52af1aa") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:48.130319 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:48.130293    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:48.130747 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:48.130445    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:48.130747 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:48.130480    2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:54:48.130747 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:48.130548    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs podName:95a07519-99bf-47ac-b236-c642491d5648 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:20.130528621 +0000 UTC m=+66.310256069 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs") pod "network-metrics-daemon-k6rks" (UID: "95a07519-99bf-47ac-b236-c642491d5648") : secret "metrics-daemon-secret" not found
Oct 30 02:54:48.134679 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:48.134651    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:48.231136 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:48.231092    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55b4z\" (UniqueName: \"kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z\") pod \"network-check-target-xg9mx\" (UID: \"23d4d339-696a-4b94-b34f-19852eb3cbc2\") " pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:48.235553 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:48.235510    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55b4z\" (UniqueName: \"kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z\") pod \"network-check-target-xg9mx\" (UID: \"23d4d339-696a-4b94-b34f-19852eb3cbc2\") " pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:48.292552 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:48.292512    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:48.347727 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:48.347689    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:48.793105 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:48.792476    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7zx8h" event={"ID":"2e4c1ab5-989e-4fe6-a028-85579a18a996","Type":"ContainerStarted","Data":"9bb8d2d71354a5a99af612879718e4dd66ae73f37a22e9a17dcc05f6472523aa"}
Oct 30 02:54:48.842058 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:48.841999    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-7zx8h" podStartSLOduration=366.687613163 podStartE2EDuration="6m10.841981498s" totalImagesPullingTime="4.154368335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:48.840337715 +0000 UTC m=+35.020065163" watchObservedRunningTime="2026-10-30 02:54:48.841981498 +0000 UTC m=+35.021708946"
Oct 30 02:54:48.842155 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:48.842123    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2" podStartSLOduration=367.660116761 podStartE2EDuration="6m9.84211451s" totalImagesPullingTime="2.181997749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:46.82085655 +0000 UTC m=+33.000583998" watchObservedRunningTime="2026-10-30 02:54:48.84211451 +0000 UTC m=+35.021841961"
Oct 30 02:54:49.014596 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.014504    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xg9mx"]
Oct 30 02:54:49.035895 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.035865    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ffnvh"]
Oct 30 02:54:49.262003 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.261962    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b4bvl"]
Oct 30 02:54:49.281480 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.281410    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.288906 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.288879    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b4bvl"]
Oct 30 02:54:49.301866 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.301841    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:54:49.302163 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.302144    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qkhcx\""
Oct 30 02:54:49.302247 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.302170    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:54:49.444107 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.444075    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zck4n\" (UniqueName: \"kubernetes.io/projected/5255451b-5afa-4272-b134-38226bfec7d0-kube-api-access-zck4n\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.444274 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.444209    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.444274 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.444249    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5255451b-5afa-4272-b134-38226bfec7d0-crio-socket\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.444394 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.444370    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.444441 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.444404    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5255451b-5afa-4272-b134-38226bfec7d0-data-volume\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.544890 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.544819    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.544890 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.544861    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5255451b-5afa-4272-b134-38226bfec7d0-crio-socket\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.545057 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.544941    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.545057 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:49.544978    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:49.545057 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.545027    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5255451b-5afa-4272-b134-38226bfec7d0-crio-socket\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.545057 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:49.545042    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:50.045027063 +0000 UTC m=+36.224754491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:49.545057 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.544986    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5255451b-5afa-4272-b134-38226bfec7d0-data-volume\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.545198 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.545097    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zck4n\" (UniqueName: \"kubernetes.io/projected/5255451b-5afa-4272-b134-38226bfec7d0-kube-api-access-zck4n\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.545198 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:49.545102    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:54:49.545250 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:49.545239    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:50.045222093 +0000 UTC m=+36.224949520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : configmap "kube-rbac-proxy" not found
Oct 30 02:54:49.545301 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.545287    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5255451b-5afa-4272-b134-38226bfec7d0-data-volume\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.572033 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.571999    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zck4n\" (UniqueName: \"kubernetes.io/projected/5255451b-5afa-4272-b134-38226bfec7d0-kube-api-access-zck4n\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:49.797057 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.796982    2544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb"]
Oct 30 02:54:49.807638 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.807598    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/0.log"
Oct 30 02:54:49.807792 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.807652    2544 generic.go:356] "Generic (PLEG): container finished" podID="68b4da6f-8901-46e0-a9c8-fe3d1bea91a7" containerID="00653b927f725d05c7e8ce62f0d58bc3aa542d3b68de64968edfae4c886df5f6" exitCode=255
Oct 30 02:54:49.808286 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.808001    2544 scope.go:122] "RemoveContainer" containerID="00653b927f725d05c7e8ce62f0d58bc3aa542d3b68de64968edfae4c886df5f6"
Oct 30 02:54:49.829336 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.829305    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ffnvh" event={"ID":"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee","Type":"ContainerStarted","Data":"ccff4aef25a7334ab7f7262fa273a3ffa98695c95228daaa1b8e4ca2d423dd86"}
Oct 30 02:54:49.829459 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.829349    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8" event={"ID":"dbb2d265-64b9-4e4f-be5b-1f00ba684673","Type":"ContainerStarted","Data":"5d255ab3a482cac9e79c367f1b8384971aee14f3937ef6d83e24cb17bb6b0f96"}
Oct 30 02:54:49.829459 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.829381    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb"]
Oct 30 02:54:49.829459 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.829408    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xg9mx" event={"ID":"23d4d339-696a-4b94-b34f-19852eb3cbc2","Type":"ContainerStarted","Data":"49016304bada867a481bcde30ba864b0f3752b45a85e63a246aa6dd220041d8e"}
Oct 30 02:54:49.829459 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.829428    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb"
Oct 30 02:54:49.829751 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.829428    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2hw2r" event={"ID":"5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0","Type":"ContainerStarted","Data":"09a1fa43cbcb2d9e810a585ca68a4aa86a65f4cbeabbd614d97a37eccb8a213a"}
Oct 30 02:54:49.829832 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.829781    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt" event={"ID":"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7","Type":"ContainerDied","Data":"00653b927f725d05c7e8ce62f0d58bc3aa542d3b68de64968edfae4c886df5f6"}
Oct 30 02:54:49.829890 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.829816    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5" event={"ID":"edcb5539-95e1-4797-9a94-0f14ffe5ca00","Type":"ContainerStarted","Data":"a8fad3600ed2464dc7314c60f70a4648901fb5ee41867c46433cd16ccb94801e"}
Oct 30 02:54:49.852218 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.852171    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8" podStartSLOduration=363.920787479 podStartE2EDuration="6m8.852154909s" totalImagesPullingTime="4.93136743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:49.851234239 +0000 UTC m=+36.030961686" watchObservedRunningTime="2026-10-30 02:54:49.852154909 +0000 UTC m=+36.031882355"
Oct 30 02:54:49.919930 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.919909    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:49.920009 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.919911    2544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:54:49.920009 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.919965    2544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qf9lt\""
Oct 30 02:54:49.946951 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.946771    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2hw2r" podStartSLOduration=1.860814064 podStartE2EDuration="6.946757516s" totalImagesPullingTime="5.085943452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:49.946042463 +0000 UTC m=+36.125769913" watchObservedRunningTime="2026-10-30 02:54:49.946757516 +0000 UTC m=+36.126484964"
Oct 30 02:54:49.950943 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:49.950908    2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vp4\" (UniqueName: \"kubernetes.io/projected/ce67816d-538e-4000-9a64-b6aff6f6aa9e-kube-api-access-m7vp4\") pod \"migrator-66df5d496b-wqdmb\" (UID: \"ce67816d-538e-4000-9a64-b6aff6f6aa9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb"
Oct 30 02:54:50.052801 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.052725    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:50.052954 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.052832    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:50.052954 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.052869    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vp4\" (UniqueName: \"kubernetes.io/projected/ce67816d-538e-4000-9a64-b6aff6f6aa9e-kube-api-access-m7vp4\") pod \"migrator-66df5d496b-wqdmb\" (UID: \"ce67816d-538e-4000-9a64-b6aff6f6aa9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb"
Oct 30 02:54:50.053301 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.053283    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:50.053380 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.053349    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:51.053335177 +0000 UTC m=+37.233062616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:50.053442 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.053397    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:54:50.053442 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.053421    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:51.053413196 +0000 UTC m=+37.233140634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : configmap "kube-rbac-proxy" not found
Oct 30 02:54:50.056537 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.056484    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5" podStartSLOduration=366.267612655 podStartE2EDuration="6m11.056466102s" totalImagesPullingTime="4.788853447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:50.053707702 +0000 UTC m=+36.233435150" watchObservedRunningTime="2026-10-30 02:54:50.056466102 +0000 UTC m=+36.236193553"
Oct 30 02:54:50.082713 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.082687    2544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vp4\" (UniqueName: \"kubernetes.io/projected/ce67816d-538e-4000-9a64-b6aff6f6aa9e-kube-api-access-m7vp4\") pod \"migrator-66df5d496b-wqdmb\" (UID: \"ce67816d-538e-4000-9a64-b6aff6f6aa9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb"
Oct 30 02:54:50.163260 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.163226    2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb"
Oct 30 02:54:50.490140 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.490061    2544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb"]
Oct 30 02:54:50.814718 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.814609    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7" event={"ID":"083a96ad-9c35-4895-be37-3b8de8089360","Type":"ContainerStarted","Data":"22f5f1268ec00162ca783147d385eaf8890e71da2740c9b69ab874f2ba7c796c"}
Oct 30 02:54:50.816371 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.816336    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xg9mx" event={"ID":"23d4d339-696a-4b94-b34f-19852eb3cbc2","Type":"ContainerStarted","Data":"fef359d9a1d70be1db204e252dbe1fdf5fdf12d9f32feaafaac2b022250edc6d"}
Oct 30 02:54:50.816498 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.816398    2544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:50.817729 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.817700    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb" event={"ID":"ce67816d-538e-4000-9a64-b6aff6f6aa9e","Type":"ContainerStarted","Data":"5bcae7057a037a168a3716ca00ea3d6df68ee8bdb7ac7e9e0c1a9b3eea05af88"}
Oct 30 02:54:50.819397 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.819379    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 02:54:50.819803 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.819786    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/0.log"
Oct 30 02:54:50.819904 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.819823    2544 generic.go:356] "Generic (PLEG): container finished" podID="68b4da6f-8901-46e0-a9c8-fe3d1bea91a7" containerID="685fb70b582de72040fa82e238af00f761a57121bd235aab96cdf9d8c1c3cec1" exitCode=255
Oct 30 02:54:50.819904 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.819898    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt" event={"ID":"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7","Type":"ContainerDied","Data":"685fb70b582de72040fa82e238af00f761a57121bd235aab96cdf9d8c1c3cec1"}
Oct 30 02:54:50.820024 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.819940    2544 scope.go:122] "RemoveContainer" containerID="00653b927f725d05c7e8ce62f0d58bc3aa542d3b68de64968edfae4c886df5f6"
Oct 30 02:54:50.820352 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.820106    2544 scope.go:122] "RemoveContainer" containerID="685fb70b582de72040fa82e238af00f761a57121bd235aab96cdf9d8c1c3cec1"
Oct 30 02:54:50.820352 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.820307    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5n6xt_openshift-console-operator(68b4da6f-8901-46e0-a9c8-fe3d1bea91a7)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt" podUID="68b4da6f-8901-46e0-a9c8-fe3d1bea91a7"
Oct 30 02:54:50.868799 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.868746    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7" podStartSLOduration=331.137395618 podStartE2EDuration="5m36.868719473s" totalImagesPullingTime="5.731323855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:50.864112206 +0000 UTC m=+37.043839652" watchObservedRunningTime="2026-10-30 02:54:50.868719473 +0000 UTC m=+37.048446926"
Oct 30 02:54:50.962149 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.962111    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:50.962149 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.962155    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:50.962382 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.962198    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:50.962382 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.962274    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:50.962382 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.962274    2544 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:50.962382 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.962290    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bbffcc788-vdsgf: secret "image-registry-tls" not found
Oct 30 02:54:50.962382 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.962295    2544 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cc9c5bc45-q8v5t: secret "image-registry-tls" not found
Oct 30 02:54:50.962382 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.962347    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls podName:ed07eb99-f3f5-4dd2-8989-970f992de3fc nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.962328145 +0000 UTC m=+45.142055588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls") pod "image-registry-7cc9c5bc45-q8v5t" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc") : secret "image-registry-tls" not found
Oct 30 02:54:50.962771 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.962396    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:50.962771 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.962504    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:50.962771 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.962564    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:50.962771 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.962664    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:50.962771 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.962766    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:50.963008 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.962794    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls podName:bd3e5117-fce3-467f-946b-745eaa13f3e7 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.96277447 +0000 UTC m=+45.142501910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls") pod "image-registry-5bbffcc788-vdsgf" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7") : secret "image-registry-tls" not found
Oct 30 02:54:50.963008 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.962284    2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:50.963008 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.962838    2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:54:50.963008 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.962846    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls podName:d921d44f-5ee7-449a-87da-2ebbb52af1aa nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.962835889 +0000 UTC m=+45.142563321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls") pod "dns-default-qggb8" (UID: "d921d44f-5ee7-449a-87da-2ebbb52af1aa") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:50.963008 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.962870    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls podName:ded308e2-be12-49e1-9ddc-153bedb23ce1 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.96286176 +0000 UTC m=+45.142589200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c2fsj" (UID: "ded308e2-be12-49e1-9ddc-153bedb23ce1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:54:50.963255 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.963072    2544 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:54:50.963255 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.963109    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert podName:c59c81e5-84cf-4660-9dd3-f3d8999812ec nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.963098938 +0000 UTC m=+45.142826377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5wtlc" (UID: "c59c81e5-84cf-4660-9dd3-f3d8999812ec") : secret "networking-console-plugin-cert" not found
Oct 30 02:54:50.963255 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.963190    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.963172446 +0000 UTC m=+45.142899875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : configmap references non-existent config key: service-ca.crt
Oct 30 02:54:50.963511 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.963486    2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:54:50.963608 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.963539    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.963524712 +0000 UTC m=+45.143252139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : secret "router-metrics-certs-default" not found
Oct 30 02:54:50.963845 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.963828    2544 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:54:50.963928 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:50.963875    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls podName:ea4de3ea-99e3-4984-bae1-5f3ae9e4460a nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.963864445 +0000 UTC m=+45.143591872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fx5sl" (UID: "ea4de3ea-99e3-4984-bae1-5f3ae9e4460a") : secret "samples-operator-tls" not found
Oct 30 02:54:50.967004 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:50.966960    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xg9mx" podStartSLOduration=36.245165963 podStartE2EDuration="36.966944942s" totalImagesPullingTime="721.778979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:50.966274328 +0000 UTC m=+37.146001776" watchObservedRunningTime="2026-10-30 02:54:50.966944942 +0000 UTC m=+37.146672390"
Oct 30 02:54:51.064117 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:51.064080    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:51.064302 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:51.064224    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:51.064302 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:51.064260    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:51.064427 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:51.064336    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:53.064319711 +0000 UTC m=+39.244047149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:51.064427 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:51.064349    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:54:51.064427 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:51.064403    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:53.064388264 +0000 UTC m=+39.244115692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : configmap "kube-rbac-proxy" not found
Oct 30 02:54:51.826061 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:51.826035    2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 02:54:51.826588 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:51.826567    2544 scope.go:122] "RemoveContainer" containerID="685fb70b582de72040fa82e238af00f761a57121bd235aab96cdf9d8c1c3cec1"
Oct 30 02:54:51.826808 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:51.826781    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5n6xt_openshift-console-operator(68b4da6f-8901-46e0-a9c8-fe3d1bea91a7)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt" podUID="68b4da6f-8901-46e0-a9c8-fe3d1bea91a7"
Oct 30 02:54:52.835229 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:52.835187    2544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb" event={"ID":"ce67816d-538e-4000-9a64-b6aff6f6aa9e","Type":"ContainerStarted","Data":"5b417e50cb5241b7093ffa5084f01e1179be7f880fca325cc22a8d989f8f9585"}
Oct 30 02:54:52.881889 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:52.881835    2544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb" podStartSLOduration=2.448386048 podStartE2EDuration="3.881820808s" totalImagesPullingTime="1.43343476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:52.880754701 +0000 UTC m=+39.060482149" watchObservedRunningTime="2026-10-30 02:54:52.881820808 +0000 UTC m=+39.061548255"
Oct 30 02:54:53.084226 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:53.084181    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:53.084386 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:53.084314    2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:53.084386 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:53.084326    2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:54:53.084456 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:53.084395    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.084373851 +0000 UTC m=+43.264101279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : configmap "kube-rbac-proxy" not found
Oct 30 02:54:53.084495 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:53.084457    2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:53.084531 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:53.084497    2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.084487838 +0000 UTC m=+43.264215276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:53.432447 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:53.432398    2544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:53.432652 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:53.432475    2544 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:53.432866 ip-10-0-140-132 kubenswrapper[2544]: I1030 02:54:53.432852    2544 scope.go:122] "RemoveContainer" containerID="685fb70b582de72040fa82e238af00f761a57121bd235aab96cdf9d8c1c3cec1"
Oct 30 02:54:53.433053 ip-10-0-140-132 kubenswrapper[2544]: E1030 02:54:53.433033    2544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5n6xt_openshift-console-operator(68b4da6f-8901-46e0-a9c8-fe3d1bea91a7)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt" podUID="68b4da6f-8901-46e0-a9c8-fe3d1bea91a7"
Oct 30 02:54:54.421855 ip-10-0-140-132 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 02:54:54.441059 ip-10-0-140-132 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 02:54:54.441216 ip-10-0-140-132 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 02:54:54.441383 ip-10-0-140-132 systemd[1]: kubelet.service: Consumed 3.119s CPU time.
Oct 30 02:54:54.459212 ip-10-0-140-132 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:54:54.540298 ip-10-0-140-132 kubenswrapper[4885]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:54.540298 ip-10-0-140-132 kubenswrapper[4885]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:54:54.540298 ip-10-0-140-132 kubenswrapper[4885]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:54.540298 ip-10-0-140-132 kubenswrapper[4885]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:54.543184 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543103    4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:54.543184 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543183    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:54.543184 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543187    4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:54.543184 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543190    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543193    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543197    4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543200    4885 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543202    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543205    4885 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543208    4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543211    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543214    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543216    4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543219    4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543222    4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543224    4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543227    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543229    4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543231    4885 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543234    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543237    4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543240    4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543242    4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:54.543358 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543245    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543248    4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543250    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543253    4885 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543255    4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543258    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543260    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543263    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543266    4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543268    4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543271    4885 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543274    4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543277    4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543279    4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543283    4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543287    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543290    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543293    4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543296    4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:54.543889 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543298    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543301    4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543305    4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543308    4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543311    4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543313    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543316    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543319    4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543322    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543324    4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543327    4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543330    4885 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543333    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543335    4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543338    4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543340    4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543343    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543346    4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543348    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:54.544361 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543351    4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543353    4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543356    4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543359    4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543361    4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543364    4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543367    4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543369    4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543372    4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543375    4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543377    4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543380    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543382    4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543385    4885 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543387    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543390    4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543392    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543395    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543397    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543401    4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:54.544855 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543403    4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543405    4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543409    4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543412    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543414    4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543417    4885 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543419    4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543422    4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543424    4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543427    4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543429    4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543432    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543434    4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543437    4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543441    4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543444    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543448    4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543451    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543454    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:54.545359 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543457    4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543459    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543462    4885 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543465    4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543467    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.543471    4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544255    4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544274    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544278    4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544282    4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544284    4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544288    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544290    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544293    4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544295    4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544298    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544302    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544305    4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544310    4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544314    4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:54.545863 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544318    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544320    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544323    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544325    4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544328    4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544330    4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544333    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544336    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544339    4885 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544342    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544344    4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544347    4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544350    4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544352    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544355    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544358    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544360    4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544365    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544368    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:54.546398 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544371    4885 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544373    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544376    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544379    4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544381    4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544384    4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544386    4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544388    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544391    4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544393    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544397    4885 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544399    4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544402    4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544404    4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544407    4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544410    4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544412    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544414    4885 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544417    4885 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544420    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:54.546912 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544423    4885 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544426    4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544428    4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544431    4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544433    4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544436    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544439    4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544441    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544443    4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544446    4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544450    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544452    4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544455    4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544457    4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544459    4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544462    4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544464    4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544467    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544469    4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:54.547404 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544472    4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544474    4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544477    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544480    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544482    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544485    4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544487    4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544491    4885 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544493    4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544496    4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544498    4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544501    4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544503    4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544506    4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544508    4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544510    4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544513    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544515    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544518    4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:54.547894 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544520    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544523    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544525    4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544527    4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544533    4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544535    4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544538    4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544540    4885 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544543    4885 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544546    4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544548    4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544553    4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544556    4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544559    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.544562    4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545133    4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545141    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545146    4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545149    4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:54.548355 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545152    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545155    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545158    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545161    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545163    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545166    4885 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545169    4885 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545171    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545174    4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545176    4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545179    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545181    4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545184    4885 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545187    4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545190    4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545192    4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545195    4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545197    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545200    4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545213    4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:54.548841 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545218    4885 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545221    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545224    4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545226    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545229    4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545233    4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545236    4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545238    4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545241    4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545243    4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545246    4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545249    4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545251    4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545254    4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545256    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545259    4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545261    4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545264    4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545266    4885 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:54.549335 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545269    4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545271    4885 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545274    4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545276    4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545281    4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545285    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545287    4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545290    4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545292    4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545295    4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545297    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545300    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545302    4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545305    4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545308    4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545312    4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545315    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545318    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545321    4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:54.549860 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545324    4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545327    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545330    4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545333    4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545335    4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545338    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545341    4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545343    4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545346    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545348    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545351    4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545353    4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545356    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545358    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545361    4885 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545363    4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545366    4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545369    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545371    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:54.550334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545374    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545376    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545379    4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545381    4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545383    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545386    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545388    4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545391    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545393    4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545396    4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545398    4885 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545400    4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545404    4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545407    4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545410    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545412    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545414    4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545417    4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545419    4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:54.550836 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545422    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545426    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545429    4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545431    4885 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545434    4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.545436    4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545527    4885 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545534    4885 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545540    4885 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545545    4885 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545550    4885 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545553    4885 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545557    4885 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545561    4885 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545564    4885 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545567    4885 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545571    4885 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545574    4885 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545577    4885 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545580    4885 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545582    4885 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545585    4885 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545592    4885 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545595    4885 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:54:54.551300 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545600    4885 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545603    4885 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545606    4885 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545625    4885 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545631    4885 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545636    4885 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545638    4885 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545642    4885 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545645    4885 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545648    4885 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545651    4885 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545654    4885 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545658    4885 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545661    4885 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545665    4885 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545668    4885 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545671    4885 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545674    4885 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545677    4885 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545680    4885 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545684    4885 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545687    4885 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545690    4885 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545693    4885 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:54:54.551891 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545696    4885 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545700    4885 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545703    4885 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545706    4885 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545709    4885 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545712    4885 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545716    4885 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545719    4885 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545722    4885 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545726    4885 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545729    4885 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545731    4885 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545736    4885 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545738    4885 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545741    4885 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545744    4885 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545747    4885 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545751    4885 flags.go:64] FLAG: --help="false"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545754    4885 flags.go:64] FLAG: --hostname-override="ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545760    4885 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545764    4885 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545767    4885 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545770    4885 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:54:54.552510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545774    4885 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545776    4885 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545779    4885 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545782    4885 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545785    4885 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545788    4885 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545791    4885 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545794    4885 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545796    4885 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545799    4885 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545802    4885 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545805    4885 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545807    4885 flags.go:64] FLAG: --lock-file=""
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545810    4885 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545813    4885 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545817    4885 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545823    4885 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545825    4885 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545830    4885 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545833    4885 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545836    4885 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545839    4885 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545842    4885 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545845    4885 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545850    4885 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:54:54.553135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545853    4885 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545857    4885 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545860    4885 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545863    4885 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545867    4885 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545871    4885 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545873    4885 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545877    4885 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545880    4885 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545887    4885 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545890    4885 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545893    4885 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545896    4885 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545899    4885 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545902    4885 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545905    4885 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545908    4885 flags.go:64] FLAG: --port="10250"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545911    4885 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545913    4885 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06a63c2ebcf15d44f"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545916    4885 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545919    4885 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545922    4885 flags.go:64] FLAG: --register-node="true"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545926    4885 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545930    4885 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545933    4885 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545936    4885 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:54:54.553774 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545939    4885 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545943    4885 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545946    4885 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545949    4885 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545952    4885 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545955    4885 flags.go:64] FLAG: --runonce="false"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545958    4885 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545961    4885 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545964    4885 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545967    4885 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545970    4885 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545973    4885 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545976    4885 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545979    4885 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545982    4885 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545984    4885 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545987    4885 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545990    4885 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545993    4885 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545996    4885 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.545998    4885 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546004    4885 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546006    4885 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546009    4885 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546014    4885 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:54:54.554421 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546017    4885 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546019    4885 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546022    4885 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546026    4885 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546029    4885 flags.go:64] FLAG: --v="2"
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546033    4885 flags.go:64] FLAG: --version="false"
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546038    4885 flags.go:64] FLAG: --vmodule=""
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546046    4885 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546049    4885 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546178    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546181    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546184    4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546188    4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546190    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546193    4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546196    4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546198    4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546201    4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546203    4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546206    4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546208    4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:54.555123 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546211    4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546214    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546216    4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546219    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546221    4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546224    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546226    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546229    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546232    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546234    4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546237    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546239    4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546241    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546244    4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546248    4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546250    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546253    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546256    4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546259    4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546261    4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:54.555772 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546264    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546267    4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546269    4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546272    4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546274    4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546277    4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546279    4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546282    4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546285    4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546287    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546290    4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546293    4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546296    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546298    4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546301    4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546303    4885 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546306    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546308    4885 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546311    4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546313    4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:54.556552 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546316    4885 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546318    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546321    4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546323    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546325    4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546328    4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546332    4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546334    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546337    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546341    4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546343    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546346    4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546362    4885 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546367    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546370    4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546374    4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546378    4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546382    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546385    4885 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:54.557168 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546388    4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546390    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546393    4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546396    4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546398    4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546401    4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546404    4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546406    4885 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546409    4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546412    4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546414    4885 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546417    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546419    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546421    4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546424    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546426    4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546428    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546431    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546433    4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546437    4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:54.557771 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546440    4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546442    4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546445    4885 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546448    4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546451    4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546453    4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546456    4885 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546458    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546461    4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546463    4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546466    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546468    4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546471    4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546473    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546475    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.546500    4885 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:54.558260 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546535    4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546539    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546543    4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546546    4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546548    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546551    4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546554    4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546557    4885 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546559    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546562    4885 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546564    4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546567    4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546569    4885 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546572    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546575    4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546577    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546582    4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546585    4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546589    4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546591    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:54.558819 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546594    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546597    4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546599    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546602    4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546604    4885 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546606    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546609    4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546629    4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546634    4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546638    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546642    4885 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546645    4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546648    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546651    4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546654    4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546657    4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546660    4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546663    4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546665    4885 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:54.559408 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546668    4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546670    4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546672    4885 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546675    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546677    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546680    4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546682    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546685    4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546687    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546692    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546694    4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546699    4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546702    4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546704    4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546707    4885 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546709    4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546712    4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546715    4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546717    4885 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546720    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:54.559967 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546722    4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546725    4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546727    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546730    4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546732    4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546735    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546737    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546740    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546742    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546745    4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546748    4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546750    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546753    4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546755    4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546758    4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546760    4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546762    4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546765    4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546767    4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:54.560462 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546770    4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546772    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546776    4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546778    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546782    4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546785    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546787    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546790    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546792    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546795    4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546797    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546800    4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546802    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546805    4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546807    4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546810    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546812    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546815    4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546817    4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546820    4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:54.561008 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546822    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546825    4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546827    4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546833    4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546836    4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546839    4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546841    4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.546843    4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.550594    4885 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.550626    4885 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.550646    4885 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.550651    4885 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550707    4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550712    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550715    4885 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550718    4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550733    4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550735    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550738    4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:54.561494 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550741    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550743    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550746    4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550748    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550751    4885 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550753    4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550756    4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550759    4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550761    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550764    4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550766    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550769    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550771    4885 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550774    4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550776    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550779    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550781    4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550784    4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550787    4885 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550789    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:54.561992 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550792    4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550795    4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550798    4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550801    4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550803    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550806    4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550808    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550810    4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550813    4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550815    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550818    4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550821    4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550823    4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550826    4885 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550828    4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550832    4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550835    4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550838    4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550843    4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:54.562469 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550846    4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550849    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550853    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550856    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550859    4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550861    4885 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550864    4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550866    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550868    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550871    4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550873    4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550876    4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550879    4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550881    4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550884    4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550887    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550890    4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550892    4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550895    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550898    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:54.562947 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550900    4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550902    4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550905    4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550908    4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550910    4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550912    4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550915    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550918    4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550920    4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550922    4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550925    4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550927    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550930    4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550932    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550935    4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550937    4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550940    4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550942    4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550945    4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:54.563418 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550947    4885 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550949    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550952    4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550954    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550957    4885 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550959    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550962    4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550965    4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550967    4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550970    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550973    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550976    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550978    4885 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550981    4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550983    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550986    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550988    4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550990    4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550992    4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:54.563895 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550995    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.550997    4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.551024    4885 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551161    4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551166    4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551169    4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551171    4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551174    4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551176    4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551179    4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551181    4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551184    4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551187    4885 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551189    4885 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551191    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551194    4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:54.564350 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551196    4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551199    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551201    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551203    4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551207    4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551209    4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551212    4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551214    4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551217    4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551220    4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551223    4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551225    4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551228    4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551230    4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551232    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551235    4885 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551237    4885 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551239    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551242    4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551244    4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:54.564763 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551246    4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551249    4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551251    4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551253    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551256    4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551258    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551261    4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551264    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551266    4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551269    4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551271    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551274    4885 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551276    4885 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551278    4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551281    4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551283    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551286    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551288    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551291    4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551293    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:54.565294 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551296    4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551300    4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551304    4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551308    4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551311    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551314    4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551317    4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551319    4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551322    4885 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551324    4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551327    4885 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551329    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551332    4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551334    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551337    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551339    4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551343    4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551347    4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551350    4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:54.565787 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551353    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551356    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551358    4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551361    4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551364    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551367    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551369    4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551371    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551374    4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551376    4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551379    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551381    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551383    4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551386    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551390    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551394    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551396    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551399    4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551401    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:54.566251 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551404    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551406    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551409    4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551412    4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551414    4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551416    4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551419    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551421    4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551424    4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551426    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551429    4885 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551431    4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551433    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551436    4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551438    4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.551460    4885 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:54.566864 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551492    4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551496    4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551499    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551502    4885 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551504    4885 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551507    4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551509    4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551512    4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551515    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551517    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551519    4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551522    4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551526    4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551531    4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551534    4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551537    4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551539    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551542    4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551544    4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551546    4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:54.567261 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551549    4885 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551551    4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551554    4885 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551556    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551559    4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551561    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551564    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551567    4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551570    4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551574    4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551576    4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551580    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551583    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551585    4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551588    4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551590    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551593    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551595    4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551598    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:54.567793 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551600    4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551603    4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551605    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551607    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551626    4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551631    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551633    4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551636    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551639    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551642    4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551644    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551647    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551650    4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551653    4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551655    4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551658    4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551660    4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551663    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551665    4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:54.568275 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551668    4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551670    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551673    4885 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551675    4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551678    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551680    4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551683    4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551685    4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551688    4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551690    4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551693    4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551695    4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551697    4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551700    4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551702    4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551705    4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551707    4885 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551710    4885 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551714    4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551716    4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:54.568781 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551719    4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551722    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551725    4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551727    4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551730    4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551732    4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551735    4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551737    4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551750    4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551754    4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551757    4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551759    4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551762    4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551765    4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551769    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551772    4885 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551775    4885 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551777    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551780    4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551783    4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:54.569296 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551785    4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551787    4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551790    4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551792    4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551795    4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551797    4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551800    4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:54.551802    4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.551858    4885 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.552834    4885 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.552889    4885 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.553071    4885 server.go:1014] "Starting client certificate rotation"
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.553219    4885 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.553275    4885 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:49:14 +0000 UTC" deadline="2028-05-15 04:32:20.013419686 +0000 UTC"
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.553319    4885 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13513h37m25.460104378s"
Oct 30 02:54:54.569801 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.553640    4885 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:54:54.570201 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.554741    4885 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:54:54.570201 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.557076    4885 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:54:54.570201 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.558735    4885 log.go:25] "Validated CRI v1 image API"
Oct 30 02:54:54.570201 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.559398    4885 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:54:54.570201 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.561241    4885 fs.go:135] Filesystem UUIDs: map[399f0473-eba9-4178-b219-e4f2de6981bc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ba937578-9349-4f84-bb84-fdccb72db4e5:/dev/nvme0n1p4]
Oct 30 02:54:54.570385 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.561256    4885 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08406e1cdab3a28fc3a51f0c5c3a03cd3669f17c9ef57373dc314d9a36bc258c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08406e1cdab3a28fc3a51f0c5c3a03cd3669f17c9ef57373dc314d9a36bc258c/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e04b37ed35dee6a9eb88ba0131903679d47401b4cd7a7dc4e5a921f400897aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e04b37ed35dee6a9eb88ba0131903679d47401b4cd7a7dc4e5a921f400897aa/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23ecde7fac698ec0024bd1926e08b12e03559b74c5193216dd96d14b6c2cd995/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23ecde7fac698ec0024bd1926e08b12e03559b74c5193216dd96d14b6c2cd995/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27e8b88a4c04f0515edd8aef914df11f5ecd87460697ba48feda0f14e059b734/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27e8b88a4c04f0515edd8aef914df11f5ecd87460697ba48feda0f14e059b734/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d04fc5c3be2d62b7033ea66d8a484795d952ab6e24c0103eb44231dd40f84dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d04fc5c3be2d62b7033ea66d8a484795d952ab6e24c0103eb44231dd40f84dd/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/315e6fc062942a3bdec1f4bc8bd87db05b7b437cabe827b9d5af393b3f336043/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/315e6fc062942a3bdec1f4bc8bd87db05b7b437cabe827b9d5af393b3f336043/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3640247c6527041991663673bdc0886b288f5d05f09e2c4eee4c87725c12f5d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3640247c6527041991663673bdc0886b288f5d05f09e2c4eee4c87725c12f5d0/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/400509b082b25f36fe9604165f78b1ab77eda3bc29553f12547af6a9bb2e9ebd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/400509b082b25f36fe9604165f78b1ab77eda3bc29553f12547af6a9bb2e9ebd/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40b04b27e2803713a170055e19d04f89b19fe802609728b89711fab94f7a71ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40b04b27e2803713a170055e19d04f89b19fe802609728b89711fab94f7a71ef/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49016304bada867a481bcde30ba864b0f3752b45a85e63a246aa6dd220041d8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49016304bada867a481bcde30ba864b0f3752b45a85e63a246aa6dd220041d8e/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/576d2749d1f7b505d76f101f7652276ce4d2e1428828ff65b8d7822a138c8010/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/576d2749d1f7b505d76f101f7652276ce4d2e1428828ff65b8d7822a138c8010/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bcae7057a037a168a3716ca00ea3d6df68ee8bdb7ac7e9e0c1a9b3eea05af88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bcae7057a037a168a3716ca00ea3d6df68ee8bdb7ac7e9e0c1a9b3eea05af88/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d1fb988c7d2fe5cd5fa417c7bdcb8f549738ff6c69864b7331919e5bb8ae5d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d1fb988c7d2fe5cd5fa417c7bdcb8f549738ff6c69864b7331919e5bb8ae5d3/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ce27297772f2101afb0562eb3ad266d8d0ad36cfa4e7e1a8dfa2bf0725585b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ce27297772f2101afb0562eb3ad266d8d0ad36cfa4e7e1a8dfa2bf0725585b2/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/753770bea8207aebe5b2731bbd5cc3d726692083dedff93f492645a56a14ec07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/753770bea8207aebe5b2731bbd5cc3d726692083dedff93f492645a56a14ec07/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bc7d9bf3be6c7bf2601c610170b8db20e511211e95acdb9077f35ed2b6c5a2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bc7d9bf3be6c7bf2601c610170b8db20e511211e95acdb9077f35ed2b6c5a2c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97f0bce6196b817ebf333efbaa33d6e3a93c4f882316b4c029ad038169470d04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97f0bce6196b817ebf333efbaa33d6e3a93c4f882316b4c029ad038169470d04/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ab991f730660e0a52c790a171b7db61a7d4bf7514ac35a2ebee23d2416666ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ab991f730660e0a52c790a171b7db61a7d4bf7514ac35a2ebee23d2416666ae/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d4692a6677681ecf4af55e75fb85429db38e9eec8ebb9bec1d9f7c466276209/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d4692a6677681ecf4af55e75fb85429db38e9eec8ebb9bec1d9f7c466276209/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b41bfb0e51691b48fb41455733a8ab7fbb8eeff910734c1496d3ecaaa534bba0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b41bfb0e51691b48fb41455733a8ab7fbb8eeff910734c1496d3ecaaa534bba0/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccff4aef25a7334ab7f7262fa273a3ffa98695c95228daaa1b8e4ca2d423dd86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccff4aef25a7334ab7f7262fa273a3ffa98695c95228daaa1b8e4ca2d423dd86/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5b3b707f2d875ae9fa9d6397f686fe0b18b0ef0005e7be19ff70beb659b50bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5b3b707f2d875ae9fa9d6397f686fe0b18b0ef0005e7be19ff70beb659b50bc/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0a45025b973a7465a6f421feb787efcdb9003fb31bdb25009faeb9b5459c19c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0a45025b973a7465a6f421feb787efcdb9003fb31bdb25009faeb9b5459c19c/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/083a96ad-9c35-4895-be37-3b8de8089360/volumes/kubernetes.io~projected/kube-api-access-w6t8d:{mountpoint:/var/lib/kubelet/pods/083a96ad-9c35-4895-be37-3b8de8089360/volumes/kubernetes.io~projected/kube-api-access-w6t8d major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23d4d339-696a-4b94-b34f-19852eb3cbc2/volumes/kubernetes.io~projected/kube-api-access-55b4z:{mountpoint:/var/lib/kubelet/pods/23d4d339-696a-4b94-b34f-19852eb3cbc2/volumes/kubernetes.io~projected/kube-api-access-55b4z major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6/volumes/kubernetes.io~projected/kube-api-access-vgdqh:{mountpoint:/var/lib/kubelet/pods/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6/volumes/kubernetes.io~projected/kube-api-access-vgdqh major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e4c1ab5-989e-4fe6-a028-85579a18a996/volumes/kubernetes.io~projected/kube-api-access-9t8tp:{mountpoint:/var/lib/kubelet/pods/2e4c1ab5-989e-4fe6-a028-85579a18a996/volumes/kubernetes.io~projected/kube-api-access-9t8tp major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e4c1ab5-989e-4fe6-a028-85579a18a996/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2e4c1ab5-989e-4fe6-a028-85579a18a996/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49a42451-a2db-456c-93ed-e0b932f13751/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/49a42451-a2db-456c-93ed-e0b932f13751/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49a42451-a2db-456c-93ed-e0b932f13751/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/49a42451-a2db-456c-93ed-e0b932f13751/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49a42451-a2db-456c-93ed-e0b932f13751/volumes/kubernetes.io~projected/kube-api-access-77q9p:{mountpoint:/var/lib/kubelet/pods/49a42451-a2db-456c-93ed-e0b932f13751/volumes/kubernetes.io~projected/kube-api-access-77q9p major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bfba258-be89-4d08-8691-1e7ccd648533/volumes/kubernetes.io~projected/kube-api-access-chclf:{mountpoint:/var/lib/kubelet/pods/4bfba258-be89-4d08-8691-1e7ccd648533/volumes/kubernetes.io~projected/kube-api-access-chclf major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5255451b-5afa-4272-b134-38226bfec7d0/volumes/kubernetes.io~projected/kube-api-access-zck4n:{mountpoint:/var/lib/kubelet/pods/5255451b-5afa-4272-b134-38226bfec7d0/volumes/kubernetes.io~projected/kube-api-access-zck4n major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0/volumes/kubernetes.io~projected/kube-api-access-q95nw:{mountpoint:/var/lib/kubelet/pods/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0/volumes/kubernetes.io~projected/kube-api-access-q95nw major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0/volumes/kubernetes.io~secret/cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f8769fd-d1f7-4e86-b367-e5cc0e707174/volumes/kubernetes.io~projected/kube-api-access-j2f9l:{mountpoint:/var/lib/kubelet/pods/5f8769fd-d1f7-4e86-b367-e5cc0e707174/volumes/kubernetes.io~projected/kube-api-access-j2f9l major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/volumes/kubernetes.io~projected/kube-api-access-gsrrk:{mountpoint:/var/lib/kubelet/pods/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/volumes/kubernetes.io~projected/kube-api-access-gsrrk major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f24e8ae-7115-48d9-b285-738049028e7b/volumes/kubernetes.io~projected/kube-api-access-pbwxs:{mountpoint:/var/lib/kubelet/pods/6f24e8ae-7115-48d9-b285-738049028e7b/volumes/kubernetes.io~projected/kube-api-access-pbwxs major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/768098d2-ba3f-41b8-8263-b74826c98c73/volumes/kubernetes.io~projected/kube-api-access-g4hqw:{mountpoint:/var/lib/kubelet/pods/768098d2-ba3f-41b8-8263-b74826c98c73/volumes/kubernetes.io~projected/kube-api-access-g4hqw major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/768098d2-ba3f-41b8-8263-b74826c98c73/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/768098d2-ba3f-41b8-8263-b74826c98c73/volumes/kubernetes.io~secret/default-certificate major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/768098d2-ba3f-41b8-8263-b74826c98c73/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/768098d2-ba3f-41b8-8263-b74826c98c73/volumes/kubernetes.io~secret/stats-auth major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ddbeb45-7185-4bc0-91de-dd2619eda1d4/volumes/kubernetes.io~projected/kube-api-access-p9r6z:{mountpoint:/var/lib/kubelet/pods/8ddbeb45-7185-4bc0-91de-dd2619eda1d4/volumes/kubernetes.io~projected/kube-api-access-p9r6z major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95a07519-99bf-47ac-b236-c642491d5648/volumes/kubernetes.io~projected/kube-api-access-s5r4v:{mountpoint:/var/lib/kubelet/pods/95a07519-99bf-47ac-b236-c642491d5648/volumes/kubernetes.io~projected/kube-api-access-s5r4v major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/971e53b3-ff90-4be0-bf64-9c1013099b9d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/971e53b3-ff90-4be0-bf64-9c1013099b9d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/971e53b3-ff90-4be0-bf64-9c1013099b9d/volumes/kubernetes.io~projected/kube-api-access-hvgls:{mountpoint:/var/lib/kubelet/pods/971e53b3-ff90-4be0-bf64-9c1013099b9d/volumes/kubernetes.io~projected/kube-api-access-hvgls major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/971e53b3-ff90-4be0-bf64-9c1013099b9d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/971e53b3-ff90-4be0-bf64-9c1013099b9d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a09a8421-0c4e-4343-90ac-a3d3b7f9ba36/volumes/kubernetes.io~projected/kube-api-access-m7q55:{mountpoint:/var/lib/kubelet/pods/a09a8421-0c4e-4343-90ac-a3d3b7f9ba36/volumes/kubernetes.io~projected/kube-api-access-m7q55 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes/kubernetes.io~projected/kube-api-access-29fjt:{mountpoint:/var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes/kubernetes.io~projected/kube-api-access-29fjt major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce67816d-538e-4000-9a64-b6aff6f6aa9e/volumes/kubernetes.io~projected/kube-api-access-m7vp4:{mountpoint:/var/lib/kubelet/pods/ce67816d-538e-4000-9a64-b6aff6f6aa9e/volumes/kubernetes.io~projected/kube-api-access-m7vp4 major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d921d44f-5ee7-449a-87da-2ebbb52af1aa/volumes/kubernetes.io~projected/kube-api-access-q85mq:{mountpoint:/var/lib/kubelet/pods/d921d44f-5ee7-449a-87da-2ebbb52af1aa/volumes/kubernetes.io~projected/kube-api-access-q85mq major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbb2d265-64b9-4e4f-be5b-1f00ba684673/volumes/kubernetes.io~projected/kube-api-access-sffkm:{mountpoint:/var/lib/kubelet/pods/dbb2d265-64b9-4e4f-be5b-1f00ba684673/volumes/kubernetes.io~projected/kube-api-access-sffkm major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbb2d265-64b9-4e4f-be5b-1f00ba684673/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/dbb2d265-64b9-4e4f-be5b-1f00ba684673/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ded308e2-be12-49e1-9ddc-153bedb23ce1/volumes/kubernetes.io~projected/kube-api-access-jdsmr:{mountpoint:/var/lib/kubelet/pods/ded308e2-be12-49e1-9ddc-153bedb23ce1/volumes/kubernetes.io~projected/kube-api-access-jdsmr major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/def369e4-1dd4-49b8-8ee7-b6b21168a589/volumes/kubernetes.io~projected/kube-api-access-qntrv:{mountpoint:/var/lib/kubelet/pods/def369e4-1dd4-49b8-8ee7-b6b21168a589/volumes/kubernetes.io~projected/kube-api-access-qntrv major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2811f6f-7a04-455e-953c-18c1018672d8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e2811f6f-7a04-455e-953c-18c1018672d8/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a/volumes/kubernetes.io~projected/kube-api-access-5p9fq:{mountpoint:/var/lib/kubelet/pods/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a/volumes/kubernetes.io~projected/kube-api-access-5p9fq major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb190b5e-126c-4d02-9c3a-35dc5c10d521/volumes/kubernetes.io~projected/kube-api-access-58r88:{mountpoint:/var/lib/kubelet/pods/eb190b5e-126c-4d02-9c3a-35dc5c10d521/volumes/kubernetes.io~projected/kube-api-access-58r88 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb32bcb9-2633-49f9-9e36-b378cf3a7b09/volumes/kubernetes.io~projected/kube-api-access-9m9gn:{mountpoint:/var/lib/kubelet/pods/eb32bcb9-2633-49f9-9e36-b378cf3a7b09/volumes/kubernetes.io~projected/kube-api-access-9m9gn major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes/kubernetes.io~projected/kube-api-access-7b2xf:{mountpoint:/var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes/kubernetes.io~projected/kube-api-access-7b2xf major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edcb5539-95e1-4797-9a94-0f14ffe5ca00/volumes/kubernetes.io~projected/kube-api-access-zq8pf:{mountpoint:/var/lib/kubelet/pods/edcb5539-95e1-4797-9a94-0f14ffe5ca00/volumes/kubernetes.io~projected/kube-api-access-zq8pf major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edcb5539-95e1-4797-9a94-0f14ffe5ca00/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/edcb5539-95e1-4797-9a94-0f14ffe5ca00/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5/volumes/kubernetes.io~projected/kube-api-access-nhzw7:{mountpoint:/var/lib/kubelet/pods/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5/volumes/kubernetes.io~projected/kube-api-access-nhzw7 major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5050adfc16b681b7b9262d8db274f8fa968801c9d2a37988f3eaea7ff67d32f2/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8738199aa1fb00223132666fb4078fc5408bacfe655c1e26de3e45dc5ab6a678/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/37a197c8a2d431d3083c5ae3d6c29a5fa8bb9ff41921607de20ccdee8233e3ae/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/14e34bd07eb5f0f0fe3116ed898a7a2db46b852df3f4b4670b62902c7a4ee1b7/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0d9c8d576490acb5e7df1bbbe0beba379f8ae5c06f34b36ba5557ce058776349/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/83e13392a897061041c7402dcff6b8dc27dc89dc51efa15d6a0dcfd12f3bf186/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6efdf2c16af932275ca40f5aeac980d41b40bc0340f5bbb155e2f7c718100918/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ac6647da30af3b3d7deacb4a53c33cdc7b0e60b2b1b9995d3e63a0c6b4462367/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/7a6459405d3ab44120580033ad11665fa6a77d42111fd272285aaeb4afb40837/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/d0670fd7ed34caae2c7e8f00e6de19c03f8e7c18163c74ae461f4d01bf3347da/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/de429697f0fb5e492fe1f9c38c95551b328be485b0cf5abd85a379fee344e0c9/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/1ca0ddda97ac080eb114dce859de45c00eacea1597fe46105fed88244265fe0f/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e9ab083ca0c251a0a5325ff87932c99bfb914a5e452e9b7b2acdb62a64cea484/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/fc7d1a77d352a5ee11c23e9dc9bca97b5cbad122ea72581852377bd10bedc329/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/b3309e9114985e9597fa393aa140ae30feadf146bf6676d731ae84547f1dccc3/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6d29c0492bdfb75eebbf9fcf6630f4c6d4e7850a5e6286af068c6a7227ebc82a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8514ecdbb9eb268799318ac3e5fe431fe385a36c5c06ed650f304e6cd528dae7/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d746e82922d0dd27b3dbcf6bbe1e678a99730d46e3967c0a6f26b4b3e26bef24/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/262d07649030d8d68e9c42d9b81d963cdcf6f2d7a103c6a31513acc3420d8ee1/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/187a4694797b1ed52f64f190635474bb282e7a1dba8f95c120493ffc5bc0fc3a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/717da37585ff69e7e86750b47fc27dba3730a15b4869f471dc79f84aab4f129b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/c1f3d35976f1c017e9820b3c90e36a9c48923d5db3f38e67c83c7d15a76d033c/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/551872c10af7f62dcbb9acde9674016f33a7e9680fa37534e0c1e8c4b5df5203/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/1afe1475175ee9d5df61215231b40176b445594226e22240ddbfc865be85f45e/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/202bbef5ee9a0921004a6463ef8fdf70ac059d97569692845725056e8fcd0458/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/95c61de2a4ef9116d1cef0abcc0c0c5f93f5d2636498088e942b162efc4e1417/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/7c160a19ba8541fbde0e363c166c38765617133c25a48e93567050bd3ab6e5af/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/c3f35a31d8dd9708357f1f645ebdd9775a9fdbac3474a526a848476e0a330b30/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/69995cbfefc92583a3f8f6817915d31c04f4ebe5b1f5d3ec1b1f6f525c1d86c5/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/29d6c381e42b758bb1a1c76c1cdaa3bdba97b7643d3fc8760551815dfbced412/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/9c7d190310b37a4ad590d55b3337b4de1e7f83a0a72964934b0610c5ddf9ef40/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/bf60843027ccc88fa22950c761f17be63d021a5686ea2eb6c9bda635208c9f78/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/44a3c38baaec4a08b2b68068ce7b71c04515d14ff42593a07a8320dbb8c7ef6d/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/865d9cd445abfd5997b37c94fa50f34fa93bed823e4ec40095f2028cd3b7a748/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/e0f9bffa8354f40310a7c72c3e36f1e369c3910e8eba056621a35e6fc31b8756/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/cdac997b33a60c693b4962c390150dc5d99320cb42aafcf8826004c47ca18111/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/4d101805986bf2cdce93e87e5ab3aefdb42d4cbc83aa82f5a08147a90b799f3e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/4f74ea54a0f57d20186fa6410f15b6fa749ef1bf9655385784a34a5b283ed966/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/86d37d34860190f4e4d70dd7e46efbceaee2fa97cfc360452f3cd5bd1ad50a49/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/0252e0cde7d293cafb6bb985812277329b8c76781c1e2640bf9eab6516a0c724/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/7ce769a61aa5c27d2c668f69d93dfe843437969b15148a056f9401623b6d5cd7/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/7cb178d535cdb95db4f2585fd60836cc20708d83c7b1d934d690206b1b73b17b/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/0d724e7f23e961049ec7a0a083879fdc62a777bcc763e3564a625bcb44241285/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/81b3445527d9cf9e690d4f5f20d762a970c9e818bddb61adcd3065fa05bb5d0d/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/9bd6f22c4bb07555ba32a8481c49591e1eb6c4bf0b40f014aaff5af9c0dc4919/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/765365faf1547ab1a3cf37c402a0b87ddd6722da0d9b5baf28130e3149f8fdb2/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/14f43510c97104c9d7d2b4ef69442c2aac9294889fd332db5c4aeabada2eb896/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/1107faccd70a5dff40b972fa6f2b9df716916c095747a09f6a2d6afe939c510f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/020a9cfafd9f47ac1facfba87db16a505db5232e571c91b1a11cfb66ff389e4e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/329e2001344b2651bb1787fb9ae70bcdcfafeb2e55507594e067211fbd8d2b49/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e5527c0f725553864fb101d608b0a94c40f116e5b5f3beddb44507b5619f1969/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/74ba7a7b5f00fd9ffef09f662d566f3e1e6e0f644892546ad45334d5cd16af05/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/3bec2b524dd143f2544a3823ac14f17b46e006d4ecc16800e8c529e7e7e126ff/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/868e03208b4294896b865dc03a03e67abe384c2e9e9f03ea18c6e5fa49107876/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e384277fb2298abe876da738e8ef622ae4e29c8b4f196d5df6e78f3ca0734ed6/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/289e9592a8196905bd87a4a94ce7ad7a23da39c3d299df729e241f42ed55dcf6/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.568497    4885 manager.go:217] Machine: {Timestamp:2026-10-30 02:54:54.568043087 +0000 UTC m=+0.078212104 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104346 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a727b0c0c79c9f293d059253588ee SystemUUID:ec2a727b-0c0c-79c9-f293-d059253588ee BootID:3c259e25-cea6-4461-9cfc-e73a99c9c63d Filesystems:[{Device:/var/lib/kubelet/pods/971e53b3-ff90-4be0-bf64-9c1013099b9d/volumes/kubernetes.io~projected/kube-api-access-hvgls DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bcae7057a037a168a3716ca00ea3d6df68ee8bdb7ac7e9e0c1a9b3eea05af88/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/eb190b5e-126c-4d02-9c3a-35dc5c10d521/volumes/kubernetes.io~projected/kube-api-access-58r88 DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/49a42451-a2db-456c-93ed-e0b932f13751/volumes/kubernetes.io~projected/kube-api-access-77q9p DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3640247c6527041991663673bdc0886b288f5d05f09e2c4eee4c87725c12f5d0/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5b3b707f2d875ae9fa9d6397f686fe0b18b0ef0005e7be19ff70beb659b50bc/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27e8b88a4c04f0515edd8aef914df11f5ecd87460697ba48feda0f14e059b734/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e04b37ed35dee6a9eb88ba0131903679d47401b4cd7a7dc4e5a921f400897aa/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/49a42451-a2db-456c-93ed-e0b932f13751/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/971e53b3-ff90-4be0-bf64-9c1013099b9d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2e4c1ab5-989e-4fe6-a028-85579a18a996/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/768098d2-ba3f-41b8-8263-b74826c98c73/volumes/kubernetes.io~projected/kube-api-access-g4hqw DeviceMajor:0 DeviceMinor:209 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/083a96ad-9c35-4895-be37-3b8de8089360/volumes/kubernetes.io~projected/kube-api-access-w6t8d DeviceMajor:0 DeviceMinor:235 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/edcb5539-95e1-4797-9a94-0f14ffe5ca00/volumes/kubernetes.io~projected/kube-api-access-zq8pf DeviceMajor:0 DeviceMinor:222 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bc7d9bf3be6c7bf2601c610170b8db20e511211e95acdb9077f35ed2b6c5a2c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:229 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d4692a6677681ecf4af55e75fb85429db38e9eec8ebb9bec1d9f7c466276209/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/23d4d339-696a-4b94-b34f-19852eb3cbc2/volumes/kubernetes.io~projected/kube-api-access-55b4z DeviceMajor:0 DeviceMinor:294 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5255451b-5afa-4272-b134-38226bfec7d0/volumes/kubernetes.io~projected/kube-api-access-zck4n DeviceMajor:0 DeviceMinor:318 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes/kubernetes.io~projected/kube-api-access-7b2xf DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49016304bada867a481bcde30ba864b0f3752b45a85e63a246aa6dd220041d8e/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f8769fd-d1f7-4e86-b367-e5cc0e707174/volumes/kubernetes.io~projected/kube-api-access-j2f9l DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0a45025b973a7465a6f421feb787efcdb9003fb31bdb25009faeb9b5459c19c/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23ecde7fac698ec0024bd1926e08b12e03559b74c5193216dd96d14b6c2cd995/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40b04b27e2803713a170055e19d04f89b19fe802609728b89711fab94f7a71ef/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/400509b082b25f36fe9604165f78b1ab77eda3bc29553f12547af6a9bb2e9ebd/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/315e6fc062942a3bdec1f4bc8bd87db05b7b437cabe827b9d5af393b3f336043/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d04fc5c3be2d62b7033ea66d8a484795d952ab6e24c0103eb44231dd40f84dd/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ab991f730660e0a52c790a171b7db61a7d4bf7514ac35a2ebee23d2416666ae/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dbb2d265-64b9-4e4f-be5b-1f00ba684673/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ded308e2-be12-49e1-9ddc-153bedb23ce1/volumes/kubernetes.io~projected/kube-api-access-jdsmr DeviceMajor:0 DeviceMinor:210 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a09a8421-0c4e-4343-90ac-a3d3b7f9ba36/volumes/kubernetes.io~projected/kube-api-access-m7q55 DeviceMajor:0 DeviceMinor:220 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:299 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb32bcb9-2633-49f9-9e36-b378cf3a7b09/volumes/kubernetes.io~projected/kube-api-access-9m9gn DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/def369e4-1dd4-49b8-8ee7-b6b21168a589/volumes/kubernetes.io~projected/kube-api-access-qntrv DeviceMajor:0 DeviceMinor:211 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/576d2749d1f7b505d76f101f7652276ce4d2e1428828ff65b8d7822a138c8010/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbb2d265-64b9-4e4f-be5b-1f00ba684673/volumes/kubernetes.io~projected/kube-api-access-sffkm DeviceMajor:0 DeviceMinor:212 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4bfba258-be89-4d08-8691-1e7ccd648533/volumes/kubernetes.io~projected/kube-api-access-chclf DeviceMajor:0 DeviceMinor:221 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/753770bea8207aebe5b2731bbd5cc3d726692083dedff93f492645a56a14ec07/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d1fb988c7d2fe5cd5fa417c7bdcb8f549738ff6c69864b7331919e5bb8ae5d3/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/49a42451-a2db-456c-93ed-e0b932f13751/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5/volumes/kubernetes.io~projected/kube-api-access-nhzw7 DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b41bfb0e51691b48fb41455733a8ab7fbb8eeff910734c1496d3ecaaa534bba0/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccff4aef25a7334ab7f7262fa273a3ffa98695c95228daaa1b8e4ca2d423dd86/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a/volumes/kubernetes.io~projected/kube-api-access-5p9fq DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0/volumes/kubernetes.io~projected/kube-api-access-q95nw DeviceMajor:0 DeviceMinor:237 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/edcb5539-95e1-4797-9a94-0f14ffe5ca00/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2e4c1ab5-989e-4fe6-a028-85579a18a996/volumes/kubernetes.io~projected/kube-api-access-9t8tp DeviceMajor:0 DeviceMinor:217 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97f0bce6196b817ebf333efbaa33d6e3a93c4f882316b4c029ad038169470d04/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ce67816d-538e-4000-9a64-b6aff6f6aa9e/volumes/kubernetes.io~projected/kube-api-access-m7vp4 DeviceMajor:0 DeviceMinor:329 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/971e53b3-ff90-4be0-bf64-9c1013099b9d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8ddbeb45-7185-4bc0-91de-dd2619eda1d4/volumes/kubernetes.io~projected/kube-api-access-p9r6z DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ce27297772f2101afb0562eb3ad266d8d0ad36cfa4e7e1a8dfa2bf0725585b2/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d921d44f-5ee7-449a-87da-2ebbb52af1aa/volumes/kubernetes.io~projected/kube-api-access-q85mq DeviceMajor:0 DeviceMinor:214 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95a07519-99bf-47ac-b236-c642491d5648/volumes/kubernetes.io~projected/kube-api-access-s5r4v DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f24e8ae-7115-48d9-b285-738049028e7b/volumes/kubernetes.io~projected/kube-api-access-pbwxs DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08406e1cdab3a28fc3a51f0c5c3a03cd3669f17c9ef57373dc314d9a36bc258c/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/768098d2-ba3f-41b8-8263-b74826c98c73/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e2811f6f-7a04-455e-953c-18c1018672d8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6/volumes/kubernetes.io~projected/kube-api-access-vgdqh DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/768098d2-ba3f-41b8-8263-b74826c98c73/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:205 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes/kubernetes.io~projected/kube-api-access-29fjt DeviceMajor:0 DeviceMinor:216 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/volumes/kubernetes.io~projected/kube-api-access-gsrrk DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:08406e1cdab3a28 MacAddress:b6:dd:4e:43:75:8c Speed:10000 Mtu:8901} {Name:27e8b88a4c04f05 MacAddress:ea:7d:4c:cb:4f:55 Speed:10000 Mtu:8901} {Name:315e6fc062942a3 MacAddress:12:35:92:88:19:ef Speed:10000 Mtu:8901} {Name:49016304bada867 MacAddress:3a:b5:d6:25:90:bd Speed:10000 Mtu:8901} {Name:5bcae7057a037a1 MacAddress:9e:7d:88:b7:a6:ea Speed:10000 Mtu:8901} {Name:5d1fb988c7d2fe5 MacAddress:a6:9f:91:3f:8c:d4 Speed:10000 Mtu:8901} {Name:753770bea8207ae MacAddress:1a:39:b7:87:c2:3e Speed:10000 Mtu:8901} {Name:97f0bce6196b817 MacAddress:0a:5e:de:3e:a7:85 Speed:10000 Mtu:8901} {Name:9d4692a6677681e MacAddress:ae:95:f2:08:93:47 Speed:10000 Mtu:8901} {Name:b41bfb0e51691b4 MacAddress:42:f3:b6:31:38:64 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e4:70:05:c9 Speed:0 Mtu:9001} {Name:br-int MacAddress:32:26:38:44:af:cf Speed:0 Mtu:8901} {Name:ccff4aef25a7334 MacAddress:2a:8c:ca:38:09:27 Speed:10000 Mtu:8901} {Name:d5b3b707f2d875a MacAddress:1a:77:11:e4:52:7a Speed:10000 Mtu:8901} {Name:e0a45025b973a74 MacAddress:02:98:5c:a2:d9:ba Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e4:70:05:c9 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:3a:58:c9:e3:24:94 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3a:00:c8:3f:0b:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.568833    4885 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.568874    4885 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569144    4885 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569163    4885 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569306    4885 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569314    4885 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569328    4885 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569349    4885 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569430    4885 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569588    4885 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569727    4885 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569765    4885 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569787    4885 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569807    4885 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569824    4885 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:54:54.570593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.569849    4885 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:54:54.571213 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.570982    4885 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:54:54.571213 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.570991    4885 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:54:54.571468 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.571445    4885 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574238    4885 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574260    4885 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574382    4885 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574692    4885 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574743    4885 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574854    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574871    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574881    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574897    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574906    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574916    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574925    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574934    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574944    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574954    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574966    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:54:54.575045 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.574981    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:54:54.575683 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.575291    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:54:54.575683 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.575310    4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:54:54.578922 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.578907    4885 server.go:1263] "Started kubelet"
Oct 30 02:54:54.579369 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.578979    4885 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:54:54.579369 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.579065    4885 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:54:54.579369 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.579137    4885 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:54:54.579549 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.579435    4885 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:54:54.579477 ip-10-0-140-132 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:54:54.581056 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.580284    4885 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:54:54.584001 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.582646    4885 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:54:54.584001 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.582665    4885 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:54:54.584001 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.582735    4885 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:49:16 +0000 UTC" deadline="2028-06-17 21:00:52.699837705 +0000 UTC"
Oct 30 02:54:54.584001 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.582768    4885 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14322h5m58.117072638s"
Oct 30 02:54:54.584001 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.582816    4885 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:54:54.584001 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.582824    4885 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:54:54.584001 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.582975    4885 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:54:54.584001 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:54.583386    4885 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-132.ec2.internal\" not found"
Oct 30 02:54:54.586299 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586263    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbb2d265-64b9-4e4f-be5b-1f00ba684673" volumeName="kubernetes.io/secret/dbb2d265-64b9-4e4f-be5b-1f00ba684673-serving-cert" seLinuxMountContext=""
Oct 30 02:54:54.586362 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586306    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3e5117-fce3-467f-946b-745eaa13f3e7" volumeName="kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-kube-api-access-29fjt" seLinuxMountContext=""
Oct 30 02:54:54.586362 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586320    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="def369e4-1dd4-49b8-8ee7-b6b21168a589" volumeName="kubernetes.io/projected/def369e4-1dd4-49b8-8ee7-b6b21168a589-kube-api-access-qntrv" seLinuxMountContext=""
Oct 30 02:54:54.586362 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586332    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed07eb99-f3f5-4dd2-8989-970f992de3fc" volumeName="kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-kube-api-access-7b2xf" seLinuxMountContext=""
Oct 30 02:54:54.586362 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586347    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed07eb99-f3f5-4dd2-8989-970f992de3fc" volumeName="kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 02:54:54.586362 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586359    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edcb5539-95e1-4797-9a94-0f14ffe5ca00" volumeName="kubernetes.io/configmap/edcb5539-95e1-4797-9a94-0f14ffe5ca00-config" seLinuxMountContext=""
Oct 30 02:54:54.586500 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586371    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0" volumeName="kubernetes.io/projected/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0-kube-api-access-q95nw" seLinuxMountContext=""
Oct 30 02:54:54.586500 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586388    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971e53b3-ff90-4be0-bf64-9c1013099b9d" volumeName="kubernetes.io/projected/971e53b3-ff90-4be0-bf64-9c1013099b9d-kube-api-access-hvgls" seLinuxMountContext=""
Oct 30 02:54:54.586500 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586402    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed07eb99-f3f5-4dd2-8989-970f992de3fc" volumeName="kubernetes.io/empty-dir/ed07eb99-f3f5-4dd2-8989-970f992de3fc-ca-trust-extracted" seLinuxMountContext=""
Oct 30 02:54:54.586500 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586414    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f8769fd-d1f7-4e86-b367-e5cc0e707174" volumeName="kubernetes.io/projected/5f8769fd-d1f7-4e86-b367-e5cc0e707174-kube-api-access-j2f9l" seLinuxMountContext=""
Oct 30 02:54:54.586500 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586424    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="768098d2-ba3f-41b8-8263-b74826c98c73" volumeName="kubernetes.io/projected/768098d2-ba3f-41b8-8263-b74826c98c73-kube-api-access-g4hqw" seLinuxMountContext=""
Oct 30 02:54:54.586500 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586435    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="768098d2-ba3f-41b8-8263-b74826c98c73" volumeName="kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-stats-auth" seLinuxMountContext=""
Oct 30 02:54:54.586500 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586446    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3e5117-fce3-467f-946b-745eaa13f3e7" volumeName="kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-installation-pull-secrets" seLinuxMountContext=""
Oct 30 02:54:54.586500 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586457    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd7651fe-9b08-41ab-b4e4-5df2fbee97ee" volumeName="kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret" seLinuxMountContext=""
Oct 30 02:54:54.586500 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586470    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed07eb99-f3f5-4dd2-8989-970f992de3fc" volumeName="kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-installation-pull-secrets" seLinuxMountContext=""
Oct 30 02:54:54.586500 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586487    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e4c1ab5-989e-4fe6-a028-85579a18a996" volumeName="kubernetes.io/projected/2e4c1ab5-989e-4fe6-a028-85579a18a996-kube-api-access-9t8tp" seLinuxMountContext=""
Oct 30 02:54:54.586500 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586498    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3e5117-fce3-467f-946b-745eaa13f3e7" volumeName="kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-trusted-ca" seLinuxMountContext=""
Oct 30 02:54:54.586865 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586509    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d921d44f-5ee7-449a-87da-2ebbb52af1aa" volumeName="kubernetes.io/configmap/d921d44f-5ee7-449a-87da-2ebbb52af1aa-config-volume" seLinuxMountContext=""
Oct 30 02:54:54.586865 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586520    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed07eb99-f3f5-4dd2-8989-970f992de3fc" volumeName="kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-certificates" seLinuxMountContext=""
Oct 30 02:54:54.586865 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586535    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edcb5539-95e1-4797-9a94-0f14ffe5ca00" volumeName="kubernetes.io/secret/edcb5539-95e1-4797-9a94-0f14ffe5ca00-serving-cert" seLinuxMountContext=""
Oct 30 02:54:54.586865 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586547    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ded308e2-be12-49e1-9ddc-153bedb23ce1" volumeName="kubernetes.io/configmap/ded308e2-be12-49e1-9ddc-153bedb23ce1-telemetry-config" seLinuxMountContext=""
Oct 30 02:54:54.586865 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586559    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" volumeName="kubernetes.io/projected/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-kube-api-access-vgdqh" seLinuxMountContext=""
Oct 30 02:54:54.586865 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586569    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5255451b-5afa-4272-b134-38226bfec7d0" volumeName="kubernetes.io/empty-dir/5255451b-5afa-4272-b134-38226bfec7d0-data-volume" seLinuxMountContext=""
Oct 30 02:54:54.586865 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586590    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f8769fd-d1f7-4e86-b367-e5cc0e707174" volumeName="kubernetes.io/configmap/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-daemon-config" seLinuxMountContext=""
Oct 30 02:54:54.586865 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586602    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68b4da6f-8901-46e0-a9c8-fe3d1bea91a7" volumeName="kubernetes.io/projected/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-kube-api-access-gsrrk" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586875    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ddbeb45-7185-4bc0-91de-dd2619eda1d4" volumeName="kubernetes.io/projected/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-kube-api-access-p9r6z" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586897    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95a07519-99bf-47ac-b236-c642491d5648" volumeName="kubernetes.io/projected/95a07519-99bf-47ac-b236-c642491d5648-kube-api-access-s5r4v" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586923    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971e53b3-ff90-4be0-bf64-9c1013099b9d" volumeName="kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-env-overrides" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.586952    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3e5117-fce3-467f-946b-745eaa13f3e7" volumeName="kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587018    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c59c81e5-84cf-4660-9dd3-f3d8999812ec" volumeName="kubernetes.io/configmap/c59c81e5-84cf-4660-9dd3-f3d8999812ec-nginx-conf" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587031    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ded308e2-be12-49e1-9ddc-153bedb23ce1" volumeName="kubernetes.io/projected/ded308e2-be12-49e1-9ddc-153bedb23ce1-kube-api-access-jdsmr" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587044    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2811f6f-7a04-455e-953c-18c1018672d8" volumeName="kubernetes.io/secret/e2811f6f-7a04-455e-953c-18c1018672d8-agent-certs" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587063    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea4de3ea-99e3-4984-bae1-5f3ae9e4460a" volumeName="kubernetes.io/projected/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-kube-api-access-5p9fq" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587075    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edcb5539-95e1-4797-9a94-0f14ffe5ca00" volumeName="kubernetes.io/projected/edcb5539-95e1-4797-9a94-0f14ffe5ca00-kube-api-access-zq8pf" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587093    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e4c1ab5-989e-4fe6-a028-85579a18a996" volumeName="kubernetes.io/configmap/2e4c1ab5-989e-4fe6-a028-85579a18a996-service-ca-bundle" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587106    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68b4da6f-8901-46e0-a9c8-fe3d1bea91a7" volumeName="kubernetes.io/secret/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-serving-cert" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587122    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f24e8ae-7115-48d9-b285-738049028e7b" volumeName="kubernetes.io/projected/6f24e8ae-7115-48d9-b285-738049028e7b-kube-api-access-pbwxs" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587149    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb190b5e-126c-4d02-9c3a-35dc5c10d521" volumeName="kubernetes.io/projected/eb190b5e-126c-4d02-9c3a-35dc5c10d521-kube-api-access-58r88" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587184    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" volumeName="kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587205    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e4c1ab5-989e-4fe6-a028-85579a18a996" volumeName="kubernetes.io/configmap/2e4c1ab5-989e-4fe6-a028-85579a18a996-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587221    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49a42451-a2db-456c-93ed-e0b932f13751" volumeName="kubernetes.io/empty-dir/49a42451-a2db-456c-93ed-e0b932f13751-tmp" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587239    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5255451b-5afa-4272-b134-38226bfec7d0" volumeName="kubernetes.io/projected/5255451b-5afa-4272-b134-38226bfec7d0-kube-api-access-zck4n" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587253    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971e53b3-ff90-4be0-bf64-9c1013099b9d" volumeName="kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovnkube-config" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587267    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce67816d-538e-4000-9a64-b6aff6f6aa9e" volumeName="kubernetes.io/projected/ce67816d-538e-4000-9a64-b6aff6f6aa9e-kube-api-access-m7vp4" seLinuxMountContext=""
Oct 30 02:54:54.587274 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587285    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f8769fd-d1f7-4e86-b367-e5cc0e707174" volumeName="kubernetes.io/configmap/5f8769fd-d1f7-4e86-b367-e5cc0e707174-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587299    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68b4da6f-8901-46e0-a9c8-fe3d1bea91a7" volumeName="kubernetes.io/configmap/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-config" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587318    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5" volumeName="kubernetes.io/projected/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-kube-api-access-nhzw7" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587332    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" volumeName="kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587351    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49a42451-a2db-456c-93ed-e0b932f13751" volumeName="kubernetes.io/empty-dir/49a42451-a2db-456c-93ed-e0b932f13751-etc-tuned" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587371    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971e53b3-ff90-4be0-bf64-9c1013099b9d" volumeName="kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587383    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d921d44f-5ee7-449a-87da-2ebbb52af1aa" volumeName="kubernetes.io/empty-dir/d921d44f-5ee7-449a-87da-2ebbb52af1aa-tmp-dir" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587400    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d921d44f-5ee7-449a-87da-2ebbb52af1aa" volumeName="kubernetes.io/projected/d921d44f-5ee7-449a-87da-2ebbb52af1aa-kube-api-access-q85mq" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587413    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2811f6f-7a04-455e-953c-18c1018672d8" volumeName="kubernetes.io/configmap/e2811f6f-7a04-455e-953c-18c1018672d8-konnectivity-ca" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587426    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb190b5e-126c-4d02-9c3a-35dc5c10d521" volumeName="kubernetes.io/configmap/eb190b5e-126c-4d02-9c3a-35dc5c10d521-iptables-alerter-script" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587444    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed07eb99-f3f5-4dd2-8989-970f992de3fc" volumeName="kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-trusted-ca" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587460    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="768098d2-ba3f-41b8-8263-b74826c98c73" volumeName="kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-default-certificate" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587495    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed07eb99-f3f5-4dd2-8989-970f992de3fc" volumeName="kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-bound-sa-token" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587514    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971e53b3-ff90-4be0-bf64-9c1013099b9d" volumeName="kubernetes.io/secret/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587528    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a09a8421-0c4e-4343-90ac-a3d3b7f9ba36" volumeName="kubernetes.io/projected/a09a8421-0c4e-4343-90ac-a3d3b7f9ba36-kube-api-access-m7q55" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587548    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3e5117-fce3-467f-946b-745eaa13f3e7" volumeName="kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-certificates" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587565    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3e5117-fce3-467f-946b-745eaa13f3e7" volumeName="kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-bound-sa-token" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587585    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5" volumeName="kubernetes.io/empty-dir/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-tmp-dir" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587598    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbb2d265-64b9-4e4f-be5b-1f00ba684673" volumeName="kubernetes.io/projected/dbb2d265-64b9-4e4f-be5b-1f00ba684673-kube-api-access-sffkm" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587633    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083a96ad-9c35-4895-be37-3b8de8089360" volumeName="kubernetes.io/projected/083a96ad-9c35-4895-be37-3b8de8089360-kube-api-access-w6t8d" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587652    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23d4d339-696a-4b94-b34f-19852eb3cbc2" volumeName="kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587681    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49a42451-a2db-456c-93ed-e0b932f13751" volumeName="kubernetes.io/projected/49a42451-a2db-456c-93ed-e0b932f13751-kube-api-access-77q9p" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587762    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bfba258-be89-4d08-8691-1e7ccd648533" volumeName="kubernetes.io/projected/4bfba258-be89-4d08-8691-1e7ccd648533-kube-api-access-chclf" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587780    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0" volumeName="kubernetes.io/secret/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0-cert" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587795    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3e5117-fce3-467f-946b-745eaa13f3e7" volumeName="kubernetes.io/empty-dir/bd3e5117-fce3-467f-946b-745eaa13f3e7-ca-trust-extracted" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587815    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbb2d265-64b9-4e4f-be5b-1f00ba684673" volumeName="kubernetes.io/configmap/dbb2d265-64b9-4e4f-be5b-1f00ba684673-config" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587832    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e4c1ab5-989e-4fe6-a028-85579a18a996" volumeName="kubernetes.io/empty-dir/2e4c1ab5-989e-4fe6-a028-85579a18a996-snapshots" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587851    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e4c1ab5-989e-4fe6-a028-85579a18a996" volumeName="kubernetes.io/secret/2e4c1ab5-989e-4fe6-a028-85579a18a996-serving-cert" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587925    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68b4da6f-8901-46e0-a9c8-fe3d1bea91a7" volumeName="kubernetes.io/configmap/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-trusted-ca" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587939    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea4de3ea-99e3-4984-bae1-5f3ae9e4460a" volumeName="kubernetes.io/configmap/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-config" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587959    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb32bcb9-2633-49f9-9e36-b378cf3a7b09" volumeName="kubernetes.io/configmap/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-serviceca" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587973    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb32bcb9-2633-49f9-9e36-b378cf3a7b09" volumeName="kubernetes.io/projected/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-kube-api-access-9m9gn" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.587992    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" volumeName="kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.588042    4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e4c1ab5-989e-4fe6-a028-85579a18a996" volumeName="kubernetes.io/empty-dir/2e4c1ab5-989e-4fe6-a028-85579a18a996-tmp" seLinuxMountContext=""
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.588056    4885 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:54:54.588280 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.588064    4885 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:54:54.591086 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.589514    4885 factory.go:55] Registering systemd factory
Oct 30 02:54:54.591086 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.589539    4885 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:54:54.591086 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.589847    4885 factory.go:153] Registering CRI-O factory
Oct 30 02:54:54.591086 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.589859    4885 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:54:54.591086 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.589910    4885 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:54:54.591086 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.589929    4885 factory.go:103] Registering Raw factory
Oct 30 02:54:54.591086 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.589943    4885 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:54:54.591086 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.590263    4885 manager.go:319] Starting recovery of all containers
Oct 30 02:54:54.609150 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.609126    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 02:54:54.609670 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.609630    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 02:54:54.616100 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.616074    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 02:54:54.616594 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.616569    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 02:54:54.616714 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.616590    4885 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:54:54.618069 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.618052    4885 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:54:54.618171 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.618092    4885 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:54:54.618171 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.618112    4885 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:54:54.618171 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:54.618147    4885 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 02:54:54.631028 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.630999    4885 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 02:54:54.631711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.631693    4885 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:54.631836 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.631717    4885 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:54.640510 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.640489    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 02:54:54.640927 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.640883    4885 generic.go:356] "Generic (PLEG): container finished" podID="971e53b3-ff90-4be0-bf64-9c1013099b9d" containerID="dd0fa76e37e94099d86c7cade33ec3c32f2c4f6c90f4551184b90baacff9e67c" exitCode=1
Oct 30 02:54:54.642258 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.642230    4885 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:54.642547 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.642532    4885 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:54.644659 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.644634    4885 generic.go:356] "Generic (PLEG): container finished" podID="d6b355a5b292f9d5c5019135ffc6b2ed" containerID="b90ed3c96f4d231839ea9e4762c63140fb5c0ad01203fb9d4e9683faf6b6ac87" exitCode=0
Oct 30 02:54:54.654695 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.654677    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 02:54:54.654784 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.654702    4885 generic.go:356] "Generic (PLEG): container finished" podID="68b4da6f-8901-46e0-a9c8-fe3d1bea91a7" containerID="685fb70b582de72040fa82e238af00f761a57121bd235aab96cdf9d8c1c3cec1" exitCode=255
Oct 30 02:54:54.658139 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.658117    4885 generic.go:356] "Generic (PLEG): container finished" podID="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" containerID="86cf573a37834005d8285b8fbf05ed4917fde5d9052403ba70d390fd9eedc8bb" exitCode=0
Oct 30 02:54:54.658139 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.658136    4885 generic.go:356] "Generic (PLEG): container finished" podID="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" containerID="0818a7040d46fd92e3aae3a2064f6276b29b036c959897a5494058dac9d22dca" exitCode=0
Oct 30 02:54:54.658293 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.658144    4885 generic.go:356] "Generic (PLEG): container finished" podID="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" containerID="350af7331f91a2b3ed306bfba8d0fe6c80dcddd64bff5a64d10ddbe8d1b9a360" exitCode=0
Oct 30 02:54:54.658293 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.658153    4885 generic.go:356] "Generic (PLEG): container finished" podID="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" containerID="c095b04604c4f8dcf61cde9a94d9a3c49a320469378e19e4078fb47d552744c4" exitCode=0
Oct 30 02:54:54.658293 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.658160    4885 generic.go:356] "Generic (PLEG): container finished" podID="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" containerID="d918cc83812d35382645a134e8e022f016e15a4f42cb43ebbe3cd5a10b422174" exitCode=0
Oct 30 02:54:54.658293 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.658166    4885 generic.go:356] "Generic (PLEG): container finished" podID="27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6" containerID="674dd5146b90b388d1280423087eb3b7c1788ce50e0fc957bec9e1188d807c0f" exitCode=0
Oct 30 02:54:54.685170 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.685126    4885 manager.go:324] Recovery completed
Oct 30 02:54:54.704449 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704425    4885 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:54:54.704449 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704442    4885 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:54:54.704559 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704462    4885 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:54:54.704641 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704604    4885 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 02:54:54.704674 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704643    4885 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 02:54:54.704674 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704668    4885 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 02:54:54.704733 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704676    4885 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 02:54:54.704733 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704684    4885 policy_none.go:50] "Start"
Oct 30 02:54:54.704733 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704695    4885 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:54:54.704733 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704707    4885 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:54:54.704848 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704838    4885 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 02:54:54.704878 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704850    4885 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 02:54:54.704878 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.704855    4885 policy_none.go:44] "Start"
Oct 30 02:54:54.709983 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.709958    4885 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:54:54.712455 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.712441    4885 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:54:54.712907 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:54.712463    4885 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:54:54.712907 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.712906    4885 server.go:87] "Starting device plugin registration server"
Oct 30 02:54:54.713137 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.713126    4885 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:54:54.713192 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.713137    4885 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:54:54.713279 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.713247    4885 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:54:54.713406 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.713366    4885 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:54:54.713522 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.713412    4885 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:54:54.713522 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.713419    4885 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:54:54.713638 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.713469    4885 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:54:54.713401355Z","UUID":"a9fcd4ea-b294-4b9a-a0d3-7a6c798aacf9","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:54:54.715109 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:54.715070    4885 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:54:54.715209 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.715134    4885 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:54:54.715209 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.715158    4885 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:54:54.718206 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.718177    4885 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"]
Oct 30 02:54:54.718836 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.718818    4885 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.718975 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.718945    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal" event={"ID":"7470c725c1ecc41c5e63c668787a572e","Type":"ContainerStarted","Data":"5bff568864da222fd48d864706d480b3853d6c2efe10af29f2e8736084712fab"}
Oct 30 02:54:54.719058 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.718982    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal" event={"ID":"7470c725c1ecc41c5e63c668787a572e","Type":"ContainerStarted","Data":"1e04b37ed35dee6a9eb88ba0131903679d47401b4cd7a7dc4e5a921f400897aa"}
Oct 30 02:54:54.719058 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.719027    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal" event={"ID":"d6b355a5b292f9d5c5019135ffc6b2ed","Type":"ContainerStarted","Data":"f812732fecc6085b8f971f812c4cd77f69b15bce7d5bfca1c18292366fc78877"}
Oct 30 02:54:54.719058 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.719047    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal" event={"ID":"d6b355a5b292f9d5c5019135ffc6b2ed","Type":"ContainerDied","Data":"b90ed3c96f4d231839ea9e4762c63140fb5c0ad01203fb9d4e9683faf6b6ac87"}
Oct 30 02:54:54.719058 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.719064    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal" event={"ID":"d6b355a5b292f9d5c5019135ffc6b2ed","Type":"ContainerStarted","Data":"23ecde7fac698ec0024bd1926e08b12e03559b74c5193216dd96d14b6c2cd995"}
Oct 30 02:54:54.719253 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.719196    4885 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.741744 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.741725    4885 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:54:54.741813 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:54.741766    4885 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-132.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.742405 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.742395    4885 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:54:54.742438 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:54.742420    4885 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.813959 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.813939    4885 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:54.817206 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.817189    4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:54.817286 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.817219    4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:54.817286 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.817232    4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-132.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:54.817286 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.817268    4885 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.831736 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.831712    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7470c725c1ecc41c5e63c668787a572e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-132.ec2.internal\" (UID: \"7470c725c1ecc41c5e63c668787a572e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.831817 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.831765    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6b355a5b292f9d5c5019135ffc6b2ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"d6b355a5b292f9d5c5019135ffc6b2ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.831817 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.831788    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6b355a5b292f9d5c5019135ffc6b2ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"d6b355a5b292f9d5c5019135ffc6b2ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.850504 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.850486    4885 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.850550 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.850537    4885 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.932513 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.932488    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6b355a5b292f9d5c5019135ffc6b2ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"d6b355a5b292f9d5c5019135ffc6b2ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.932513 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.932515    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6b355a5b292f9d5c5019135ffc6b2ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"d6b355a5b292f9d5c5019135ffc6b2ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.932680 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.932533    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7470c725c1ecc41c5e63c668787a572e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-132.ec2.internal\" (UID: \"7470c725c1ecc41c5e63c668787a572e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.932680 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.932583    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6b355a5b292f9d5c5019135ffc6b2ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"d6b355a5b292f9d5c5019135ffc6b2ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.932680 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.932591    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6b355a5b292f9d5c5019135ffc6b2ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" (UID: \"d6b355a5b292f9d5c5019135ffc6b2ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:54.932680 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:54.932639    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7470c725c1ecc41c5e63c668787a572e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-132.ec2.internal\" (UID: \"7470c725c1ecc41c5e63c668787a572e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:55.570559 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.570477    4885 apiserver.go:52] "Watching apiserver"
Oct 30 02:54:55.673869 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.673837    4885 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:55.674020 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.673939    4885 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:55.702368 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.702337    4885 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:54:55.702501 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.702385    4885 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-132.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:55.702501 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.702344    4885 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:54:55.702501 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.702457    4885 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal"
Oct 30 02:54:55.702812 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.702790    4885 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:54:55.703406 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.703383    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-k6c24","openshift-image-registry/node-ca-stg5z","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl","openshift-multus/multus-fbpcs","openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb","openshift-image-registry/image-registry-7cc9c5bc45-q8v5t","openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj","kube-system/kas-connection-checker-685895c7c9-pkft5","kube-system/global-pull-secret-syncer-ffnvh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t","openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5","openshift-multus/network-metrics-daemon-k6rks","openshift-network-diagnostics/network-check-target-xg9mx","openshift-cluster-node-tuning-operator/tuned-l6rrk","openshift-console-operator/console-operator-7cd56b8dbc-5n6xt","openshift-insights/insights-operator-7c4986d449-7zx8h","openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7","openshift-network-operator/iptables-alerter-6th7l","kube-system/kube-apiserver-proxy-ip-10-0-140-132.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-244kd","kube-system/kas-connection-checker-685895c7c9-zvmbf","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2","openshift-ingress/router-default-57bd894fd7-n6trx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-132.ec2.internal","openshift-image-registry/image-registry-5bbffcc788-vdsgf","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8","openshift-ingress-canary/ingress-canary-2hw2r","kube-system/kas-connection-checker-685895c7c9-6qkm5","kube-system/konnectivity-agent-x5wjw","openshift-dns/dns-default-qggb8","openshift-insights/insights-runtime-extractor-b4bvl","openshift-multus/multus-additional-cni-plugins-hlhb7","openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"]
Oct 30 02:54:55.705197 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.705179    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:55.705748 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.705719    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.706203 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.706189    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:55.707225 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.707204    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:55.710146 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.710124    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:55.713788 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.713769    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:55.716865 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.716846    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.718213 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.718197    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:55.719342 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.719324    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.719454 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.719436    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 02:54:55.719514 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.719445    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 02:54:55.719654 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.719643    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 02:54:55.722884 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.722868    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 02:54:55.724005 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.723990    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 02:54:55.724517 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.724505    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 02:54:55.724677 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.724665    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 02:54:55.725805 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.725790    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:54:55.725805 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.725797    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.725931 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.725919    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.726626 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.726599    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 02:54:55.726668 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.726633    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.731094 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.731080    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.731428 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.731408    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.731540 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.731524    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.731722 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.731709    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.731797 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.731780    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 02:54:55.731853 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.731802    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 02:54:55.731853 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.731833    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.731853 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.731842    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.731959 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.731854    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 02:54:55.731959 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.731897    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.732201 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.732188    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:54:55.732356 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.732344    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.734859 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.734845    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.736033 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.736018    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.736196 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.736171    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 02:54:55.736263 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.736248    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 02:54:55.736688 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.736673    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.736769 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.736729    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.736825 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.736764    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4tlgk\""
Oct 30 02:54:55.736825 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.736766    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.736951 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.736937    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.737000 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.736960    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.737000 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.736963    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l729x\""
Oct 30 02:54:55.737083 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.736987    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.737083 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.737017    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:54:55.737083 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.737029    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.737083 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.737062    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 02:54:55.737083 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.737068    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 02:54:55.737083 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.737070    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.737336 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.737303    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.737384 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.737346    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:54:55.737384 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.737349    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.737451 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.737381    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:54:55.737645 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.737634    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-sdsc6\""
Oct 30 02:54:55.741578 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.741559    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tm8xk\""
Oct 30 02:54:55.741578 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.741574    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-h9jxb\""
Oct 30 02:54:55.741714 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.741583    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvbgx\""
Oct 30 02:54:55.741714 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.741589    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j9c5h\""
Oct 30 02:54:55.741714 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.741574    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:54:55.742361 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.742350    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:54:55.742457 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.742444    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qzv89\""
Oct 30 02:54:55.747126 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.747112    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k6c24_eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5/dns-node-resolver/0.log"
Oct 30 02:54:55.747253 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.747239    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 02:54:55.748593 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.748570    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 02:54:55.748693 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.748602    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.748693 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.748643    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 02:54:55.749106 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.749074    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 02:54:55.753031 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.752960    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.753031 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.752964    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 02:54:55.753031 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.752996    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.753031 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753010    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.753031 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753018    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.753031 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753032    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753116    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753125    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753140    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753158    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753164    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753184    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753184    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753204    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753205    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753219    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753235    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753242    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:54:55.753419 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753261    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.754014 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.753990    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:54:55.754695 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.754667    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:54:55.754805 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.754760    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:54:55.755068 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755054    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qkhcx\""
Oct 30 02:54:55.755139 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755131    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vnlvm\""
Oct 30 02:54:55.755201 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755191    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cr2zp\""
Oct 30 02:54:55.755299 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755287    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:54:55.755364 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755325    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:54:55.755438 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755376    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vxw8k\""
Oct 30 02:54:55.755438 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755407    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:54:55.755554 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755450    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:54:55.755554 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755479    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:54:55.755554 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755514    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d6p9p\""
Oct 30 02:54:55.755737 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755560    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z4s4b\""
Oct 30 02:54:55.755737 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755608    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f9m7f\""
Oct 30 02:54:55.755737 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755676    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:54:55.755737 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755705    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:54:55.755974 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755764    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rg86w\""
Oct 30 02:54:55.755974 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.755788    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:54:55.756349 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.756333    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 02:54:55.756418 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.756372    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 02:54:55.756418 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.756412    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-hx2qx\""
Oct 30 02:54:55.756528 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.756469    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qf9lt\""
Oct 30 02:54:55.756528 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.756515    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5k8rj\""
Oct 30 02:54:55.785437 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.785415    4885 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:54:55.836160 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836093    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-certificates\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.836160 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836122    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49a42451-a2db-456c-93ed-e0b932f13751-etc-tuned\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.836160 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836142    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffkm\" (UniqueName: \"kubernetes.io/projected/dbb2d265-64b9-4e4f-be5b-1f00ba684673-kube-api-access-sffkm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:55.836160 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836158    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed07eb99-f3f5-4dd2-8989-970f992de3fc-ca-trust-extracted\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.836453 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836186    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-installation-pull-secrets\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.836453 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836213    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a42451-a2db-456c-93ed-e0b932f13751-tmp\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.836453 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836238    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-conf-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.836453 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836274    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49a42451-a2db-456c-93ed-e0b932f13751-etc-tuned\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.836453 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836293    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-certificates\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.836453 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836313    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49a42451-a2db-456c-93ed-e0b932f13751-tmp\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.836453 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836322    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed07eb99-f3f5-4dd2-8989-970f992de3fc-ca-trust-extracted\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.836453 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836325    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8pf\" (UniqueName: \"kubernetes.io/projected/edcb5539-95e1-4797-9a94-0f14ffe5ca00-kube-api-access-zq8pf\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:55.836453 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836412    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-stats-auth\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:55.836453 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836427    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-installation-pull-secrets\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.836453 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836443    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836473    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0-cert\") pod \"ingress-canary-2hw2r\" (UID: \"5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0\") " pod="openshift-ingress-canary/ingress-canary-2hw2r"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836510    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-cni-multus\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836545    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-image-registry-private-configuration\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836574    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-sys-fs\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836601    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836675    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-stats-auth\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836679    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-var-lib-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836731    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0-cert\") pod \"ingress-canary-2hw2r\" (UID: \"5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0\") " pod="openshift-ingress-canary/ingress-canary-2hw2r"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836745    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836776    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-hostroot\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836858    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5255451b-5afa-4272-b134-38226bfec7d0-crio-socket\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836868    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-image-registry-private-configuration\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836904    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.836961 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836942    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29fjt\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-kube-api-access-29fjt\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.836999    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-certificates\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837060    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837063    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-certificates\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837057    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbb2d265-64b9-4e4f-be5b-1f00ba684673-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837120    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hqw\" (UniqueName: \"kubernetes.io/projected/768098d2-ba3f-41b8-8263-b74826c98c73-kube-api-access-g4hqw\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837148    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-kubelet\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837174    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837201    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysctl-d\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837194    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbb2d265-64b9-4e4f-be5b-1f00ba684673-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837258    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-daemon-config\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837293    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-installation-pull-secrets\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837325    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85mq\" (UniqueName: \"kubernetes.io/projected/d921d44f-5ee7-449a-87da-2ebbb52af1aa-kube-api-access-q85mq\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837352    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb190b5e-126c-4d02-9c3a-35dc5c10d521-host-slash\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837380    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837418    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-tmp-dir\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837477    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4c1ab5-989e-4fe6-a028-85579a18a996-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837506    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vp4\" (UniqueName: \"kubernetes.io/projected/ce67816d-538e-4000-9a64-b6aff6f6aa9e-kube-api-access-m7vp4\") pod \"migrator-66df5d496b-wqdmb\" (UID: \"ce67816d-538e-4000-9a64-b6aff6f6aa9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837522    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-netns\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837526    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-tmp-dir\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:55.837538 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837536    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-etc-kubernetes\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837479    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-installation-pull-secrets\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837562    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c59c81e5-84cf-4660-9dd3-f3d8999812ec-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837593    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-daemon-config\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837634    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837667    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m9gn\" (UniqueName: \"kubernetes.io/projected/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-kube-api-access-9m9gn\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837684    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzw7\" (UniqueName: \"kubernetes.io/projected/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-kube-api-access-nhzw7\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837701    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-var-lib-kubelet\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837726    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837752    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-node-log\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837773    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-run-ovn-kubernetes\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837783    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c59c81e5-84cf-4660-9dd3-f3d8999812ec-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837831    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7q55\" (UniqueName: \"kubernetes.io/projected/a09a8421-0c4e-4343-90ac-a3d3b7f9ba36-kube-api-access-m7q55\") pod \"volume-data-source-validator-58c9f5f46d-7fbf2\" (UID: \"a09a8421-0c4e-4343-90ac-a3d3b7f9ba36\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837887    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysconfig\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837841    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4c1ab5-989e-4fe6-a028-85579a18a996-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837906    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-systemd\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837921    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-host\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.837978    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-run-netns\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838008    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-multus-certs\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838033    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-bound-sa-token\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838078    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntrv\" (UniqueName: \"kubernetes.io/projected/def369e4-1dd4-49b8-8ee7-b6b21168a589-kube-api-access-qntrv\") pod \"kas-connection-checker-685895c7c9-zvmbf\" (UID: \"def369e4-1dd4-49b8-8ee7-b6b21168a589\") " pod="kube-system/kas-connection-checker-685895c7c9-zvmbf"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838153    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q95nw\" (UniqueName: \"kubernetes.io/projected/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0-kube-api-access-q95nw\") pod \"ingress-canary-2hw2r\" (UID: \"5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0\") " pod="openshift-ingress-canary/ingress-canary-2hw2r"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838173    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f8769fd-d1f7-4e86-b367-e5cc0e707174-cni-binary-copy\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838188    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55b4z\" (UniqueName: \"kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z\") pod \"network-check-target-xg9mx\" (UID: \"23d4d339-696a-4b94-b34f-19852eb3cbc2\") " pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838202    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edcb5539-95e1-4797-9a94-0f14ffe5ca00-config\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838217    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2f9l\" (UniqueName: \"kubernetes.io/projected/5f8769fd-d1f7-4e86-b367-e5cc0e707174-kube-api-access-j2f9l\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838239    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-lib-modules\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838264    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edcb5539-95e1-4797-9a94-0f14ffe5ca00-serving-cert\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838283    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-env-overrides\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838300    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsrrk\" (UniqueName: \"kubernetes.io/projected/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-kube-api-access-gsrrk\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838322    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-cnibin\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.838324 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838347    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-trusted-ca\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838369    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-log-socket\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838393    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4c1ab5-989e-4fe6-a028-85579a18a996-serving-cert\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838399    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f8769fd-d1f7-4e86-b367-e5cc0e707174-cni-binary-copy\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838417    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-sys\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838440    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-serviceca\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838471    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58r88\" (UniqueName: \"kubernetes.io/projected/eb190b5e-126c-4d02-9c3a-35dc5c10d521-kube-api-access-58r88\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838494    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-cni-netd\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838520    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovnkube-config\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838554    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwxs\" (UniqueName: \"kubernetes.io/projected/6f24e8ae-7115-48d9-b285-738049028e7b-kube-api-access-pbwxs\") pod \"kas-connection-checker-685895c7c9-pkft5\" (UID: \"6f24e8ae-7115-48d9-b285-738049028e7b\") " pod="kube-system/kas-connection-checker-685895c7c9-pkft5"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838561    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-env-overrides\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838586    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838609    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edcb5539-95e1-4797-9a94-0f14ffe5ca00-serving-cert\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838659    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edcb5539-95e1-4797-9a94-0f14ffe5ca00-config\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838676    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-system-cni-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838724    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-trusted-ca\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838738    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovnkube-config\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838765    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-device-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838785    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4c1ab5-989e-4fe6-a028-85579a18a996-serving-cert\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838810    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77q9p\" (UniqueName: \"kubernetes.io/projected/49a42451-a2db-456c-93ed-e0b932f13751-kube-api-access-77q9p\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838835    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-systemd\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838930    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvgls\" (UniqueName: \"kubernetes.io/projected/971e53b3-ff90-4be0-bf64-9c1013099b9d-kube-api-access-hvgls\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.838960    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839100    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-image-registry-private-configuration\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839133    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839160    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839173    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839206    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839236    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2811f6f-7a04-455e-953c-18c1018672d8-konnectivity-ca\") pod \"konnectivity-agent-x5wjw\" (UID: \"e2811f6f-7a04-455e-953c-18c1018672d8\") " pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839262    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-etc-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839298    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-image-registry-private-configuration\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839292    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4c1ab5-989e-4fe6-a028-85579a18a996-service-ca-bundle\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839338    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-cni-bin\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839364    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdsmr\" (UniqueName: \"kubernetes.io/projected/ded308e2-be12-49e1-9ddc-153bedb23ce1-kube-api-access-jdsmr\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839392    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-registration-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839404    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-serviceca\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839430    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-original-pull-secret\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839446    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-kubelet-config\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839451    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e4c1ab5-989e-4fe6-a028-85579a18a996-service-ca-bundle\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839477    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-dbus\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839508    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6t8d\" (UniqueName: \"kubernetes.io/projected/083a96ad-9c35-4895-be37-3b8de8089360-kube-api-access-w6t8d\") pod \"network-check-source-6f78d59f99-zq8h7\" (UID: \"083a96ad-9c35-4895-be37-3b8de8089360\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839479    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2811f6f-7a04-455e-953c-18c1018672d8-konnectivity-ca\") pod \"konnectivity-agent-x5wjw\" (UID: \"e2811f6f-7a04-455e-953c-18c1018672d8\") " pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839535    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-default-certificate\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839563    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cnibin\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839588    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysctl-conf\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839641    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-run\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839673    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2811f6f-7a04-455e-953c-18c1018672d8-agent-certs\") pod \"konnectivity-agent-x5wjw\" (UID: \"e2811f6f-7a04-455e-953c-18c1018672d8\") " pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:55.839711 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839707    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p9fq\" (UniqueName: \"kubernetes.io/projected/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-kube-api-access-5p9fq\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839797    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-default-certificate\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839797    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839880    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839906    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chclf\" (UniqueName: \"kubernetes.io/projected/4bfba258-be89-4d08-8691-1e7ccd648533-kube-api-access-chclf\") pod \"kas-connection-checker-685895c7c9-6qkm5\" (UID: \"4bfba258-be89-4d08-8691-1e7ccd648533\") " pod="kube-system/kas-connection-checker-685895c7c9-6qkm5"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839924    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ded308e2-be12-49e1-9ddc-153bedb23ce1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839936    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2811f6f-7a04-455e-953c-18c1018672d8-agent-certs\") pod \"konnectivity-agent-x5wjw\" (UID: \"e2811f6f-7a04-455e-953c-18c1018672d8\") " pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839954    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.839981    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-socket-dir-parent\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840007    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd3e5117-fce3-467f-946b-745eaa13f3e7-ca-trust-extracted\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840030    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840059    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840114    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840129    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdqh\" (UniqueName: \"kubernetes.io/projected/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-kube-api-access-vgdqh\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840151    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd3e5117-fce3-467f-946b-745eaa13f3e7-ca-trust-extracted\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840159    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-serving-cert\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840131    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ded308e2-be12-49e1-9ddc-153bedb23ce1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840193    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8tp\" (UniqueName: \"kubernetes.io/projected/2e4c1ab5-989e-4fe6-a028-85579a18a996-kube-api-access-9t8tp\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840215    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-trusted-ca\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840229    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-bound-sa-token\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840236    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840244    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-socket-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840282    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5255451b-5afa-4272-b134-38226bfec7d0-data-volume\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840310    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-trusted-ca\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840334    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-k8s-cni-cncf-io\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840360    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9r6z\" (UniqueName: \"kubernetes.io/projected/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-kube-api-access-p9r6z\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840384    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb190b5e-126c-4d02-9c3a-35dc5c10d521-iptables-alerter-script\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840391    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5255451b-5afa-4272-b134-38226bfec7d0-data-volume\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840408    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-ovn\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840443    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-host\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840501    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-hosts-file\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840531    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e4c1ab5-989e-4fe6-a028-85579a18a996-snapshots\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840560    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbb2d265-64b9-4e4f-be5b-1f00ba684673-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840596    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-trusted-ca\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840598    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-serving-cert\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840606    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovnkube-script-lib\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840708    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840742    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-config\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840740    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb190b5e-126c-4d02-9c3a-35dc5c10d521-iptables-alerter-script\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840751    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e4c1ab5-989e-4fe6-a028-85579a18a996-snapshots\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840758    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbb2d265-64b9-4e4f-be5b-1f00ba684673-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840793    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e4c1ab5-989e-4fe6-a028-85579a18a996-tmp\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840810    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovnkube-script-lib\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840854    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b2xf\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-kube-api-access-7b2xf\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840871    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e4c1ab5-989e-4fe6-a028-85579a18a996-tmp\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840899    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-modprobe-d\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840983    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-kubernetes\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841013    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.840985    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-config\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841030    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-systemd-units\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841057    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-slash\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841072    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-cni-bin\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841098    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-system-cni-dir\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841105    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-trusted-ca\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841112    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-cni-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841151    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-kubelet\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841188    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zck4n\" (UniqueName: \"kubernetes.io/projected/5255451b-5afa-4272-b134-38226bfec7d0-kube-api-access-zck4n\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841206    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovn-node-metrics-cert\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841250    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5r4v\" (UniqueName: \"kubernetes.io/projected/95a07519-99bf-47ac-b236-c642491d5648-kube-api-access-s5r4v\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841294    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841391    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d921d44f-5ee7-449a-87da-2ebbb52af1aa-config-volume\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841420    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d921d44f-5ee7-449a-87da-2ebbb52af1aa-tmp-dir\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841425    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/971e53b3-ff90-4be0-bf64-9c1013099b9d-ovn-node-metrics-cert\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841548    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d921d44f-5ee7-449a-87da-2ebbb52af1aa-tmp-dir\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:55.841750 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.841578    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d921d44f-5ee7-449a-87da-2ebbb52af1aa-config-volume\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:55.859069 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.859046    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffkm\" (UniqueName: \"kubernetes.io/projected/dbb2d265-64b9-4e4f-be5b-1f00ba684673-kube-api-access-sffkm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8\" (UID: \"dbb2d265-64b9-4e4f-be5b-1f00ba684673\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8"
Oct 30 02:54:55.864526 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.864508    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m9gn\" (UniqueName: \"kubernetes.io/projected/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-kube-api-access-9m9gn\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:55.865654 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.865590    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chclf\" (UniqueName: \"kubernetes.io/projected/4bfba258-be89-4d08-8691-1e7ccd648533-kube-api-access-chclf\") pod \"kas-connection-checker-685895c7c9-6qkm5\" (UID: \"4bfba258-be89-4d08-8691-1e7ccd648533\") " pod="kube-system/kas-connection-checker-685895c7c9-6qkm5"
Oct 30 02:54:55.865654 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.865641    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzw7\" (UniqueName: \"kubernetes.io/projected/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-kube-api-access-nhzw7\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:55.865752 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.865653    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntrv\" (UniqueName: \"kubernetes.io/projected/def369e4-1dd4-49b8-8ee7-b6b21168a589-kube-api-access-qntrv\") pod \"kas-connection-checker-685895c7c9-zvmbf\" (UID: \"def369e4-1dd4-49b8-8ee7-b6b21168a589\") " pod="kube-system/kas-connection-checker-685895c7c9-zvmbf"
Oct 30 02:54:55.865752 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.865665    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29fjt\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-kube-api-access-29fjt\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.865752 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.865687    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58r88\" (UniqueName: \"kubernetes.io/projected/eb190b5e-126c-4d02-9c3a-35dc5c10d521-kube-api-access-58r88\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:55.865843 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.865834    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdsmr\" (UniqueName: \"kubernetes.io/projected/ded308e2-be12-49e1-9ddc-153bedb23ce1-kube-api-access-jdsmr\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:55.865913 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.865897    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hqw\" (UniqueName: \"kubernetes.io/projected/768098d2-ba3f-41b8-8263-b74826c98c73-kube-api-access-g4hqw\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:55.865944 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.865926    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbwxs\" (UniqueName: \"kubernetes.io/projected/6f24e8ae-7115-48d9-b285-738049028e7b-kube-api-access-pbwxs\") pod \"kas-connection-checker-685895c7c9-pkft5\" (UID: \"6f24e8ae-7115-48d9-b285-738049028e7b\") " pod="kube-system/kas-connection-checker-685895c7c9-pkft5"
Oct 30 02:54:55.869021 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.868968    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77q9p\" (UniqueName: \"kubernetes.io/projected/49a42451-a2db-456c-93ed-e0b932f13751-kube-api-access-77q9p\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.869116 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.869037    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q95nw\" (UniqueName: \"kubernetes.io/projected/5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0-kube-api-access-q95nw\") pod \"ingress-canary-2hw2r\" (UID: \"5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0\") " pod="openshift-ingress-canary/ingress-canary-2hw2r"
Oct 30 02:54:55.869116 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.869073    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9r6z\" (UniqueName: \"kubernetes.io/projected/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-kube-api-access-p9r6z\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.869194 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.869147    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdqh\" (UniqueName: \"kubernetes.io/projected/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-kube-api-access-vgdqh\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.869336 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.869318    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p9fq\" (UniqueName: \"kubernetes.io/projected/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-kube-api-access-5p9fq\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:55.869779 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.869758    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2f9l\" (UniqueName: \"kubernetes.io/projected/5f8769fd-d1f7-4e86-b367-e5cc0e707174-kube-api-access-j2f9l\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.869870 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.869848    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7q55\" (UniqueName: \"kubernetes.io/projected/a09a8421-0c4e-4343-90ac-a3d3b7f9ba36-kube-api-access-m7q55\") pod \"volume-data-source-validator-58c9f5f46d-7fbf2\" (UID: \"a09a8421-0c4e-4343-90ac-a3d3b7f9ba36\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7fbf2"
Oct 30 02:54:55.870115 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.870097    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vp4\" (UniqueName: \"kubernetes.io/projected/ce67816d-538e-4000-9a64-b6aff6f6aa9e-kube-api-access-m7vp4\") pod \"migrator-66df5d496b-wqdmb\" (UID: \"ce67816d-538e-4000-9a64-b6aff6f6aa9e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wqdmb"
Oct 30 02:54:55.870181 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.870146    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-bound-sa-token\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.870255 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.870239    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsrrk\" (UniqueName: \"kubernetes.io/projected/68b4da6f-8901-46e0-a9c8-fe3d1bea91a7-kube-api-access-gsrrk\") pod \"console-operator-7cd56b8dbc-5n6xt\" (UID: \"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:54:55.870434 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.870416    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55b4z\" (UniqueName: \"kubernetes.io/projected/23d4d339-696a-4b94-b34f-19852eb3cbc2-kube-api-access-55b4z\") pod \"network-check-target-xg9mx\" (UID: \"23d4d339-696a-4b94-b34f-19852eb3cbc2\") " pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:54:55.871444 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.871427    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b2xf\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-kube-api-access-7b2xf\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.874327 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.874313    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-bound-sa-token\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.874385 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.874370    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8pf\" (UniqueName: \"kubernetes.io/projected/edcb5539-95e1-4797-9a94-0f14ffe5ca00-kube-api-access-zq8pf\") pod \"service-ca-operator-544d749c8c-wn4f5\" (UID: \"edcb5539-95e1-4797-9a94-0f14ffe5ca00\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5"
Oct 30 02:54:55.875239 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.875223    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8tp\" (UniqueName: \"kubernetes.io/projected/2e4c1ab5-989e-4fe6-a028-85579a18a996-kube-api-access-9t8tp\") pod \"insights-operator-7c4986d449-7zx8h\" (UID: \"2e4c1ab5-989e-4fe6-a028-85579a18a996\") " pod="openshift-insights/insights-operator-7c4986d449-7zx8h"
Oct 30 02:54:55.875956 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.875939    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85mq\" (UniqueName: \"kubernetes.io/projected/d921d44f-5ee7-449a-87da-2ebbb52af1aa-kube-api-access-q85mq\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:55.876256 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.876238    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5r4v\" (UniqueName: \"kubernetes.io/projected/95a07519-99bf-47ac-b236-c642491d5648-kube-api-access-s5r4v\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:55.876372 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.876350    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvgls\" (UniqueName: \"kubernetes.io/projected/971e53b3-ff90-4be0-bf64-9c1013099b9d-kube-api-access-hvgls\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.876434 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.876420    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6t8d\" (UniqueName: \"kubernetes.io/projected/083a96ad-9c35-4895-be37-3b8de8089360-kube-api-access-w6t8d\") pod \"network-check-source-6f78d59f99-zq8h7\" (UID: \"083a96ad-9c35-4895-be37-3b8de8089360\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zq8h7"
Oct 30 02:54:55.877916 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.877897    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zck4n\" (UniqueName: \"kubernetes.io/projected/5255451b-5afa-4272-b134-38226bfec7d0-kube-api-access-zck4n\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.942242 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942214    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-socket-dir-parent\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.942242 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942242    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.942379 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942259    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-socket-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.942379 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942285    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-k8s-cni-cncf-io\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.942379 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942335    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-socket-dir-parent\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.942379 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942342    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-socket-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.942379 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942367    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.942555 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942394    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-ovn\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.942555 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942424    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-host\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:55.942555 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942425    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-k8s-cni-cncf-io\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.942555 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942439    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-hosts-file\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:55.942555 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942454    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-ovn\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.942555 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942487    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb32bcb9-2633-49f9-9e36-b378cf3a7b09-host\") pod \"node-ca-stg5z\" (UID: \"eb32bcb9-2633-49f9-9e36-b378cf3a7b09\") " pod="openshift-image-registry/node-ca-stg5z"
Oct 30 02:54:55.942555 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942502    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5-hosts-file\") pod \"node-resolver-k6c24\" (UID: \"eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5\") " pod="openshift-dns/node-resolver-k6c24"
Oct 30 02:54:55.942555 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942533    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.942555 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942553    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-modprobe-d\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942567    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-kubernetes\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942582    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942596    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-systemd-units\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942624    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-slash\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942647    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-cni-bin\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942652    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-kubernetes\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942668    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942681    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-systemd-units\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942689    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-cni-bin\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942702    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-modprobe-d\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942708    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-slash\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942717    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-system-cni-dir\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942684    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-system-cni-dir\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942748    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-cni-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942775    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-kubelet\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942796    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-kubelet\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.942797    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:56.442773677 +0000 UTC m=+1.952942695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : configmap references non-existent config key: service-ca.crt
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942816    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-cni-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942842    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942877    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-conf-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942905    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942932    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-multus-conf-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942931    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-cni-multus\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942957    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-sys-fs\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.942966    4885 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942973    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:55.942964 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.942978    4885 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bbffcc788-vdsgf: secret "image-registry-tls" not found
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943012    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-var-lib-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943020    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-sys-fs\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943029    4885 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943050    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-hostroot\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943055    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls podName:bd3e5117-fce3-467f-946b-745eaa13f3e7 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:56.443044021 +0000 UTC m=+1.953213059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls") pod "image-registry-5bbffcc788-vdsgf" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7") : secret "image-registry-tls" not found
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943073    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-hostroot\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.942967    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-cni-multus\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943083    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls podName:d921d44f-5ee7-449a-87da-2ebbb52af1aa nodeName:}" failed. No retries permitted until 2026-10-30 02:54:56.443069425 +0000 UTC m=+1.953238449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls") pod "dns-default-qggb8" (UID: "d921d44f-5ee7-449a-87da-2ebbb52af1aa") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943030    4885 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943106    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-var-lib-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943147    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:56.443135709 +0000 UTC m=+1.953304732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : secret "router-metrics-certs-default" not found
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943151    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5255451b-5afa-4272-b134-38226bfec7d0-crio-socket\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943109    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5255451b-5afa-4272-b134-38226bfec7d0-crio-socket\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943177    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943194    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-kubelet\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943207    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943221    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysctl-d\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943237    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb190b5e-126c-4d02-9c3a-35dc5c10d521-host-slash\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943252    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943276    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-kubelet\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943280    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-netns\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943314    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943316    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-netns\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943317    4885 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943346    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-etc-kubernetes\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943313    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-etc-kubernetes\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943318    4885 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943374    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls podName:ea4de3ea-99e3-4984-bae1-5f3ae9e4460a nodeName:}" failed. No retries permitted until 2026-10-30 02:54:56.443361532 +0000 UTC m=+1.953530553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fx5sl" (UID: "ea4de3ea-99e3-4984-bae1-5f3ae9e4460a") : secret "samples-operator-tls" not found
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943383    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb190b5e-126c-4d02-9c3a-35dc5c10d521-host-slash\") pod \"iptables-alerter-6th7l\" (UID: \"eb190b5e-126c-4d02-9c3a-35dc5c10d521\") " pod="openshift-network-operator/iptables-alerter-6th7l"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943411    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs podName:95a07519-99bf-47ac-b236-c642491d5648 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:56.443403047 +0000 UTC m=+1.953572052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs") pod "network-metrics-daemon-k6rks" (UID: "95a07519-99bf-47ac-b236-c642491d5648") : secret "metrics-daemon-secret" not found
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943408    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysctl-d\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943426    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943445    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-var-lib-kubelet\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943461    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943493    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-node-log\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943515    4885 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943516    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943524    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-run-ovn-kubernetes\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943529    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-node-log\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943497    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-var-lib-kubelet\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943538    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:56.443531281 +0000 UTC m=+1.953700285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943559    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-run-ovn-kubernetes\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943577    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysconfig\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943596    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-systemd\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943638    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-host\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943660    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-run-netns\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943655    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysconfig\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943684    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-systemd\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943686    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-multus-certs\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943702    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-run-netns\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943712    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-host\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943716    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-run-multus-certs\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943720    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-lib-modules\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943775    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-cnibin\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943798    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-log-socket\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943815    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-sys\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943830    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-cni-netd\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943830    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-cnibin\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943847    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943870    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-system-cni-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943879    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-log-socket\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943890    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-cni-netd\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943848    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-lib-modules\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943879    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-sys\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943894    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-device-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943918    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-system-cni-dir\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.943886 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943936    4885 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943941    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-device-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943950    4885 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cc9c5bc45-q8v5t: secret "image-registry-tls" not found
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943963    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-systemd\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.943988    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls podName:ed07eb99-f3f5-4dd2-8989-970f992de3fc nodeName:}" failed. No retries permitted until 2026-10-30 02:54:56.44397498 +0000 UTC m=+1.954143996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls") pod "image-registry-7cc9c5bc45-q8v5t" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc") : secret "image-registry-tls" not found
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.943935    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-run-systemd\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944030    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944057    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944084    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-etc-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944127    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-etc-openvswitch\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944156    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-cni-bin\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944184    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-registration-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944210    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-kubelet-config\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944234    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-dbus\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944264    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cnibin\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944272    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f8769fd-d1f7-4e86-b367-e5cc0e707174-host-var-lib-cni-bin\") pod \"multus-fbpcs\" (UID: \"5f8769fd-d1f7-4e86-b367-e5cc0e707174\") " pod="openshift-multus/multus-fbpcs"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944273    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ddbeb45-7185-4bc0-91de-dd2619eda1d4-registration-dir\") pod \"aws-ebs-csi-driver-node-kw42t\" (UID: \"8ddbeb45-7185-4bc0-91de-dd2619eda1d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw42t"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944288    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysctl-conf\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944292    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-kubelet-config\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944323    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-run\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944327    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd7651fe-9b08-41ab-b4e4-5df2fbee97ee-dbus\") pod \"global-pull-secret-syncer-ffnvh\" (UID: \"cd7651fe-9b08-41ab-b4e4-5df2fbee97ee\") " pod="kube-system/global-pull-secret-syncer-ffnvh"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944352    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944371    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-run\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944379    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944409    4885 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944426    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971e53b3-ff90-4be0-bf64-9c1013099b9d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-244kd\" (UID: \"971e53b3-ff90-4be0-bf64-9c1013099b9d\") " pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.944429    4885 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:55.944483    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert podName:c59c81e5-84cf-4660-9dd3-f3d8999812ec nodeName:}" failed. No retries permitted until 2026-10-30 02:54:56.444468423 +0000 UTC m=+1.954637446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5wtlc" (UID: "c59c81e5-84cf-4660-9dd3-f3d8999812ec") : secret "networking-console-plugin-cert" not found
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944414    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49a42451-a2db-456c-93ed-e0b932f13751-etc-sysctl-conf\") pod \"tuned-l6rrk\" (UID: \"49a42451-a2db-456c-93ed-e0b932f13751\") " pod="openshift-cluster-node-tuning-operator/tuned-l6rrk"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944509    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5255451b-5afa-4272-b134-38226bfec7d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:55.946132 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.944538    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6-cnibin\") pod \"multus-additional-cni-plugins-hlhb7\" (UID: \"27ba9b8b-d3ef-4d0b-8efa-2d52701fc8b6\") " pod="openshift-multus/multus-additional-cni-plugins-hlhb7"
Oct 30 02:54:55.946936 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:55.946657    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ded308e2-be12-49e1-9ddc-153bedb23ce1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c2fsj\" (UID: \"ded308e2-be12-49e1-9ddc-153bedb23ce1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:56.007304 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.007284    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"
Oct 30 02:54:56.010427 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.010195    4885 scope.go:122] "RemoveContainer" containerID="685fb70b582de72040fa82e238af00f761a57121bd235aab96cdf9d8c1c3cec1"
Oct 30 02:54:56.299809 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.299777    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj"]
Oct 30 02:54:56.304012 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:56.303984    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded308e2_be12_49e1_9ddc_153bedb23ce1.slice/crio-e10fdcd68ea758cacb1738e9fb16412dc06eb9959ac46c4747b1973c7f3f45a7 WatchSource:0}: Error finding container e10fdcd68ea758cacb1738e9fb16412dc06eb9959ac46c4747b1973c7f3f45a7: Status 404 returned error can't find the container with id e10fdcd68ea758cacb1738e9fb16412dc06eb9959ac46c4747b1973c7f3f45a7
Oct 30 02:54:56.306048 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.306028    4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:54:56.448429 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.448397    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:56.448592 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.448443    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:56.448592 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.448466    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:56.448592 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.448488    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:56.448592 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.448512    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:56.448592 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.448545    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:56.448592 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448561    4885 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:54:56.448592 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448573    4885 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:56.448592 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448591    4885 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bbffcc788-vdsgf: secret "image-registry-tls" not found
Oct 30 02:54:56.448592 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448590    4885 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448578    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.448560244 +0000 UTC m=+2.958729270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : configmap references non-existent config key: service-ca.crt
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448641    4885 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448668    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.44865184 +0000 UTC m=+2.958820851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : secret "router-metrics-certs-default" not found
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.448664    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448683    4885 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448689    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls podName:bd3e5117-fce3-467f-946b-745eaa13f3e7 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.448679046 +0000 UTC m=+2.958848052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls") pod "image-registry-5bbffcc788-vdsgf" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7") : secret "image-registry-tls" not found
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448713    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs podName:95a07519-99bf-47ac-b236-c642491d5648 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.448705265 +0000 UTC m=+2.958874269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs") pod "network-metrics-daemon-k6rks" (UID: "95a07519-99bf-47ac-b236-c642491d5648") : secret "metrics-daemon-secret" not found
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448730    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls podName:d921d44f-5ee7-449a-87da-2ebbb52af1aa nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.448722223 +0000 UTC m=+2.958891235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls") pod "dns-default-qggb8" (UID: "d921d44f-5ee7-449a-87da-2ebbb52af1aa") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448748    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert podName:c59c81e5-84cf-4660-9dd3-f3d8999812ec nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.448739255 +0000 UTC m=+2.958908282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5wtlc" (UID: "c59c81e5-84cf-4660-9dd3-f3d8999812ec") : secret "networking-console-plugin-cert" not found
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.448781    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.448834    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448887    4885 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448917    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.448906931 +0000 UTC m=+2.959075960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448937    4885 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:54:56.448940 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448949    4885 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cc9c5bc45-q8v5t: secret "image-registry-tls" not found
Oct 30 02:54:56.449444 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:56.448990    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls podName:ed07eb99-f3f5-4dd2-8989-970f992de3fc nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.44897945 +0000 UTC m=+2.959148454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls") pod "image-registry-7cc9c5bc45-q8v5t" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc") : secret "image-registry-tls" not found
Oct 30 02:54:56.451197 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.451167    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4de3ea-99e3-4984-bae1-5f3ae9e4460a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fx5sl\" (UID: \"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:56.608693 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.608596    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"
Oct 30 02:54:56.678630 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.678588    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj" event={"ID":"ded308e2-be12-49e1-9ddc-153bedb23ce1","Type":"ContainerStarted","Data":"e10fdcd68ea758cacb1738e9fb16412dc06eb9959ac46c4747b1973c7f3f45a7"}
Oct 30 02:54:56.680718 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.680697    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 02:54:56.680847 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.680807    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt" event={"ID":"68b4da6f-8901-46e0-a9c8-fe3d1bea91a7","Type":"ContainerStarted","Data":"0455ac8debf2c8f874a1e3de61289ca484a5c5656c1f7c0f9fa5920a736240fe"}
Oct 30 02:54:56.746761 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.746739    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-stg5z_eb32bcb9-2633-49f9-9e36-b378cf3a7b09/node-ca/0.log"
Oct 30 02:54:56.905742 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:56.905668    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl"]
Oct 30 02:54:57.347875 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.347842    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2hw2r_5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0/serve-healthcheck-canary/0.log"
Oct 30 02:54:57.460221 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.460184    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:57.460388 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.460240    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:57.460388 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.460276    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:57.460388 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.460303    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:57.460388 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.460334    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:57.460388 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:57.460375    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:59.460352937 +0000 UTC m=+4.970521941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : configmap references non-existent config key: service-ca.crt
Oct 30 02:54:57.460672 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:57.460426    4885 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:54:57.460672 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.460429    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:57.460672 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:57.460475    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs podName:95a07519-99bf-47ac-b236-c642491d5648 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:59.460461701 +0000 UTC m=+4.970630722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs") pod "network-metrics-daemon-k6rks" (UID: "95a07519-99bf-47ac-b236-c642491d5648") : secret "metrics-daemon-secret" not found
Oct 30 02:54:57.460672 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:57.460517    4885 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:57.460672 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.460522    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:57.460672 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:57.460555    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:59.460544674 +0000 UTC m=+4.970713684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:57.460672 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.460598    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:57.460672 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:57.460662    4885 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:57.461031 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:57.460702    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls podName:d921d44f-5ee7-449a-87da-2ebbb52af1aa nodeName:}" failed. No retries permitted until 2026-10-30 02:54:59.460689036 +0000 UTC m=+4.970858063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls") pod "dns-default-qggb8" (UID: "d921d44f-5ee7-449a-87da-2ebbb52af1aa") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:57.461031 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:57.460729    4885 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:54:57.461031 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:57.460769    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert podName:c59c81e5-84cf-4660-9dd3-f3d8999812ec nodeName:}" failed. No retries permitted until 2026-10-30 02:54:59.460758039 +0000 UTC m=+4.970927048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5wtlc" (UID: "c59c81e5-84cf-4660-9dd3-f3d8999812ec") : secret "networking-console-plugin-cert" not found
Oct 30 02:54:57.461031 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:57.460815    4885 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:54:57.461031 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:57.460851    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:59.460839736 +0000 UTC m=+4.971008748 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : secret "router-metrics-certs-default" not found
Oct 30 02:54:57.464057 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.464031    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") pod \"image-registry-5bbffcc788-vdsgf\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") " pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:57.476897 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.476866    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") pod \"image-registry-7cc9c5bc45-q8v5t\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") " pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:57.507029 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.507001    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:54:57.518169 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.518141    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:54:57.693979 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.693887    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl" event={"ID":"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a","Type":"ContainerStarted","Data":"b0ef2c637f8e3b81b987be4c8e02f1efe56cd04c9e5915828d62bfa24ceb1243"}
Oct 30 02:54:57.837722 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.837685    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bbffcc788-vdsgf"]
Oct 30 02:54:57.840258 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.840227    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"]
Oct 30 02:54:57.841334 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:57.841227    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3e5117_fce3_467f_946b_745eaa13f3e7.slice/crio-2b3ca2e43186722dfcc108b1452810c416fc44390af680ebab500f6f439fd057 WatchSource:0}: Error finding container 2b3ca2e43186722dfcc108b1452810c416fc44390af680ebab500f6f439fd057: Status 404 returned error can't find the container with id 2b3ca2e43186722dfcc108b1452810c416fc44390af680ebab500f6f439fd057
Oct 30 02:54:57.852749 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:57.852703    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded07eb99_f3f5_4dd2_8989_970f992de3fc.slice/crio-235fa1b42555a5ef39e9a2ffaa755472310b94d6d526ec1fc281cd479c18a648 WatchSource:0}: Error finding container 235fa1b42555a5ef39e9a2ffaa755472310b94d6d526ec1fc281cd479c18a648: Status 404 returned error can't find the container with id 235fa1b42555a5ef39e9a2ffaa755472310b94d6d526ec1fc281cd479c18a648
Oct 30 02:54:57.962379 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:57.962349    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:54:58.147382 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:58.147356    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-wqdmb_ce67816d-538e-4000-9a64-b6aff6f6aa9e/migrator/0.log"
Oct 30 02:54:58.348882 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:58.348801    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8_dbb2d265-64b9-4e4f-be5b-1f00ba684673/kube-storage-version-migrator-operator/0.log"
Oct 30 02:54:58.699820 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:58.699731    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl" event={"ID":"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a","Type":"ContainerStarted","Data":"f2c14605a905ba557697e8f4290e8539654b816a452a6dae5790bf9eae4dbec4"}
Oct 30 02:54:58.699820 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:58.699804    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fx5sl" event={"ID":"ea4de3ea-99e3-4984-bae1-5f3ae9e4460a","Type":"ContainerStarted","Data":"8093c9d5a9e844bdef606e454e5979597be3cd928365a5f57f7ea60275a07753"}
Oct 30 02:54:58.701437 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:58.701406    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t" event={"ID":"ed07eb99-f3f5-4dd2-8989-970f992de3fc","Type":"ContainerStarted","Data":"7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98"}
Oct 30 02:54:58.701559 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:58.701448    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t" event={"ID":"ed07eb99-f3f5-4dd2-8989-970f992de3fc","Type":"ContainerStarted","Data":"235fa1b42555a5ef39e9a2ffaa755472310b94d6d526ec1fc281cd479c18a648"}
Oct 30 02:54:58.703008 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:58.702984    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c2fsj" event={"ID":"ded308e2-be12-49e1-9ddc-153bedb23ce1","Type":"ContainerStarted","Data":"3f5bf3021df22d0784bd3d283904f5effb7659d7d808f11e45f792f60cb16cad"}
Oct 30 02:54:58.704600 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:58.704576    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf" event={"ID":"bd3e5117-fce3-467f-946b-745eaa13f3e7","Type":"ContainerStarted","Data":"f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c"}
Oct 30 02:54:58.704713 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:58.704627    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf" event={"ID":"bd3e5117-fce3-467f-946b-745eaa13f3e7","Type":"ContainerStarted","Data":"2b3ca2e43186722dfcc108b1452810c416fc44390af680ebab500f6f439fd057"}
Oct 30 02:54:59.483066 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.483031    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:59.483240 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.483087    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:54:59.483240 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.483120    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:54:59.483240 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.483142    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:54:59.483240 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.483164    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:54:59.483240 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:59.483234    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:03.483212566 +0000 UTC m=+8.993381589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : configmap references non-existent config key: service-ca.crt
Oct 30 02:54:59.483508 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:59.483239    4885 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:59.483508 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:59.483253    4885 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:54:59.483508 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:59.483281    4885 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:59.483508 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:59.483282    4885 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:54:59.483508 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:59.483284    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls podName:d921d44f-5ee7-449a-87da-2ebbb52af1aa nodeName:}" failed. No retries permitted until 2026-10-30 02:55:03.483273818 +0000 UTC m=+8.993442875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls") pod "dns-default-qggb8" (UID: "d921d44f-5ee7-449a-87da-2ebbb52af1aa") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:59.483508 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:59.483376    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:03.483356295 +0000 UTC m=+8.993525301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : secret "router-metrics-certs-default" not found
Oct 30 02:54:59.483508 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:59.483398    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:03.483385542 +0000 UTC m=+8.993554553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:59.483508 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.483428    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:59.483783 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:54:59.483512    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs podName:95a07519-99bf-47ac-b236-c642491d5648 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:03.483493576 +0000 UTC m=+8.993662581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs") pod "network-metrics-daemon-k6rks" (UID: "95a07519-99bf-47ac-b236-c642491d5648") : secret "metrics-daemon-secret" not found
Oct 30 02:54:59.486071 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.486050    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c59c81e5-84cf-4660-9dd3-f3d8999812ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5wtlc\" (UID: \"c59c81e5-84cf-4660-9dd3-f3d8999812ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:59.611185 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.611114    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"
Oct 30 02:54:59.711788 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.711754    4885 generic.go:356] "Generic (PLEG): container finished" podID="edcb5539-95e1-4797-9a94-0f14ffe5ca00" containerID="a8fad3600ed2464dc7314c60f70a4648901fb5ee41867c46433cd16ccb94801e" exitCode=0
Oct 30 02:54:59.712212 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.711815    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5" event={"ID":"edcb5539-95e1-4797-9a94-0f14ffe5ca00","Type":"ContainerDied","Data":"a8fad3600ed2464dc7314c60f70a4648901fb5ee41867c46433cd16ccb94801e"}
Oct 30 02:54:59.712212 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.711894    4885 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:54:59.712212 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.711905    4885 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:54:59.713109 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.713088    4885 scope.go:122] "RemoveContainer" containerID="a8fad3600ed2464dc7314c60f70a4648901fb5ee41867c46433cd16ccb94801e"
Oct 30 02:54:59.898758 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:54:59.898690    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5wtlc"]
Oct 30 02:54:59.902024 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:54:59.901992    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59c81e5_84cf_4660_9dd3_f3d8999812ec.slice/crio-79044aa563a8894c235c79983d92d9d1e4dbabc535091263027c57e79a3b65ba WatchSource:0}: Error finding container 79044aa563a8894c235c79983d92d9d1e4dbabc535091263027c57e79a3b65ba: Status 404 returned error can't find the container with id 79044aa563a8894c235c79983d92d9d1e4dbabc535091263027c57e79a3b65ba
Oct 30 02:55:00.354200 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:00.354162    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:55:00.354389 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:00.354323    4885 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:55:00.359253 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:00.359226    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5n6xt"
Oct 30 02:55:00.718465 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:00.718423    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc" event={"ID":"c59c81e5-84cf-4660-9dd3-f3d8999812ec","Type":"ContainerStarted","Data":"79044aa563a8894c235c79983d92d9d1e4dbabc535091263027c57e79a3b65ba"}
Oct 30 02:55:00.720602 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:00.720570    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wn4f5" event={"ID":"edcb5539-95e1-4797-9a94-0f14ffe5ca00","Type":"ContainerStarted","Data":"d85b219bbfa30b594385fd5c694fabbd86d3885df878b95d78cefee7a89753df"}
Oct 30 02:55:01.725350 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:01.725316    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5wtlc" event={"ID":"c59c81e5-84cf-4660-9dd3-f3d8999812ec","Type":"ContainerStarted","Data":"132f45550d601504fa162c48d1a71249b3cea63b72dd73572de0e70293c2587c"}
Oct 30 02:55:01.981931 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:01.981847    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:55:01.982062 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:01.982044    4885 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:55:02.868777 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:02.868734    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:55:03.517042 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:03.517007    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:55:03.517042 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:03.517056    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:55:03.517254 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:03.517074    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:55:03.517254 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:03.517103    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:55:03.517254 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:03.517126    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:55:03.517254 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:55:03.517194    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle podName:768098d2-ba3f-41b8-8263-b74826c98c73 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:11.517171616 +0000 UTC m=+17.027340637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle") pod "router-default-57bd894fd7-n6trx" (UID: "768098d2-ba3f-41b8-8263-b74826c98c73") : configmap references non-existent config key: service-ca.crt
Oct 30 02:55:03.517254 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:55:03.517242    4885 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:03.517430 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:55:03.517272    4885 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:55:03.517430 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:55:03.517287    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls podName:d921d44f-5ee7-449a-87da-2ebbb52af1aa nodeName:}" failed. No retries permitted until 2026-10-30 02:55:11.51727668 +0000 UTC m=+17.027445697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls") pod "dns-default-qggb8" (UID: "d921d44f-5ee7-449a-87da-2ebbb52af1aa") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:03.517430 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:55:03.517242    4885 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:03.517430 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:55:03.517319    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs podName:95a07519-99bf-47ac-b236-c642491d5648 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:11.51730562 +0000 UTC m=+17.027474643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs") pod "network-metrics-daemon-k6rks" (UID: "95a07519-99bf-47ac-b236-c642491d5648") : secret "metrics-daemon-secret" not found
Oct 30 02:55:03.517430 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:55:03.517340    4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls podName:5255451b-5afa-4272-b134-38226bfec7d0 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:11.517330688 +0000 UTC m=+17.027499695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b4bvl" (UID: "5255451b-5afa-4272-b134-38226bfec7d0") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:03.519590 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:03.519569    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768098d2-ba3f-41b8-8263-b74826c98c73-metrics-certs\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:55:04.089065 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:04.089027    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:55:07.303508 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:07.303469    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:55:08.803299 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:08.803266    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"]
Oct 30 02:55:11.585606 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.585572    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:55:11.585606 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.585608    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:55:11.586044 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.585655    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:55:11.586044 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.585712    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:55:11.586427 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.586364    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/768098d2-ba3f-41b8-8263-b74826c98c73-service-ca-bundle\") pod \"router-default-57bd894fd7-n6trx\" (UID: \"768098d2-ba3f-41b8-8263-b74826c98c73\") " pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:55:11.588087 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.588058    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d921d44f-5ee7-449a-87da-2ebbb52af1aa-metrics-tls\") pod \"dns-default-qggb8\" (UID: \"d921d44f-5ee7-449a-87da-2ebbb52af1aa\") " pod="openshift-dns/dns-default-qggb8"
Oct 30 02:55:11.588166 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.588118    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95a07519-99bf-47ac-b236-c642491d5648-metrics-certs\") pod \"network-metrics-daemon-k6rks\" (UID: \"95a07519-99bf-47ac-b236-c642491d5648\") " pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:55:11.588166 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.588150    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5255451b-5afa-4272-b134-38226bfec7d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4bvl\" (UID: \"5255451b-5afa-4272-b134-38226bfec7d0\") " pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:55:11.606354 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.606329    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:55:11.614194 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.614172    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k6rks"
Oct 30 02:55:11.619147 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.619128    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qggb8"
Oct 30 02:55:11.620269 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.620251    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b4bvl"
Oct 30 02:55:11.926343 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.926277    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k6rks"]
Oct 30 02:55:11.929719 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:55:11.929692    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a07519_99bf_47ac_b236_c642491d5648.slice/crio-80dc8e853c76fa4af64968b75dd7d5dfad2e0fce0a6b983279629e2aca10adc1 WatchSource:0}: Error finding container 80dc8e853c76fa4af64968b75dd7d5dfad2e0fce0a6b983279629e2aca10adc1: Status 404 returned error can't find the container with id 80dc8e853c76fa4af64968b75dd7d5dfad2e0fce0a6b983279629e2aca10adc1
Oct 30 02:55:11.932192 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:55:11.932164    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768098d2_ba3f_41b8_8263_b74826c98c73.slice/crio-15c2158d3640a281f56671380831f82cc4304f2e49283a9f4b2fe7415ea52bff WatchSource:0}: Error finding container 15c2158d3640a281f56671380831f82cc4304f2e49283a9f4b2fe7415ea52bff: Status 404 returned error can't find the container with id 15c2158d3640a281f56671380831f82cc4304f2e49283a9f4b2fe7415ea52bff
Oct 30 02:55:11.936379 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.936356    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-57bd894fd7-n6trx"]
Oct 30 02:55:11.941197 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.941178    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b4bvl"]
Oct 30 02:55:11.945994 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:11.945972    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qggb8"]
Oct 30 02:55:11.954376 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:55:11.954356    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd921d44f_5ee7_449a_87da_2ebbb52af1aa.slice/crio-833c69f0fdcbcdfff1f9976414966bab74af76e0b7b7b0d867bcfa3d174eb7ac WatchSource:0}: Error finding container 833c69f0fdcbcdfff1f9976414966bab74af76e0b7b7b0d867bcfa3d174eb7ac: Status 404 returned error can't find the container with id 833c69f0fdcbcdfff1f9976414966bab74af76e0b7b7b0d867bcfa3d174eb7ac
Oct 30 02:55:12.763566 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:12.763423    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4bvl" event={"ID":"5255451b-5afa-4272-b134-38226bfec7d0","Type":"ContainerStarted","Data":"7162b4f1703135b1c51b273eb89c7114adb3028194ec6bd8c1509be05b60f24c"}
Oct 30 02:55:12.763566 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:12.763498    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4bvl" event={"ID":"5255451b-5afa-4272-b134-38226bfec7d0","Type":"ContainerStarted","Data":"216a3b7eb0af386174c0e9917323a59f248990f895b3883cd91d7d28de6f0e44"}
Oct 30 02:55:12.763566 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:12.763528    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4bvl" event={"ID":"5255451b-5afa-4272-b134-38226bfec7d0","Type":"ContainerStarted","Data":"beb76bb6ffb80f0e1bff4d30032c2947be992ab6994270ff51ecbd4481331a59"}
Oct 30 02:55:12.764955 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:12.764926    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qggb8" event={"ID":"d921d44f-5ee7-449a-87da-2ebbb52af1aa","Type":"ContainerStarted","Data":"833c69f0fdcbcdfff1f9976414966bab74af76e0b7b7b0d867bcfa3d174eb7ac"}
Oct 30 02:55:12.766773 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:12.766701    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-57bd894fd7-n6trx" event={"ID":"768098d2-ba3f-41b8-8263-b74826c98c73","Type":"ContainerStarted","Data":"d90ae586799117a0faa2f7c60e7a76881cc3b3ef8f6ffba39ba9f0cc593d4fb4"}
Oct 30 02:55:12.766773 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:12.766747    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-57bd894fd7-n6trx" event={"ID":"768098d2-ba3f-41b8-8263-b74826c98c73","Type":"ContainerStarted","Data":"15c2158d3640a281f56671380831f82cc4304f2e49283a9f4b2fe7415ea52bff"}
Oct 30 02:55:12.767864 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:12.767836    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k6rks" event={"ID":"95a07519-99bf-47ac-b236-c642491d5648","Type":"ContainerStarted","Data":"80dc8e853c76fa4af64968b75dd7d5dfad2e0fce0a6b983279629e2aca10adc1"}
Oct 30 02:55:13.606663 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:13.606627    4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:55:13.609259 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:13.609237    4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:55:13.775827 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:13.775228    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k6rks" event={"ID":"95a07519-99bf-47ac-b236-c642491d5648","Type":"ContainerStarted","Data":"2eef208fa2d0c52fcb67306f29f2ada364d99ebc92c70118e4b88f350ef7b429"}
Oct 30 02:55:13.775827 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:13.775298    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k6rks" event={"ID":"95a07519-99bf-47ac-b236-c642491d5648","Type":"ContainerStarted","Data":"ec8afaab01e88bb1cc4224b93e94b8a01e6e40de9b846a4249ca20837ed55d2d"}
Oct 30 02:55:13.778423 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:13.778331    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qggb8" event={"ID":"d921d44f-5ee7-449a-87da-2ebbb52af1aa","Type":"ContainerStarted","Data":"8eef497cff3c95bb929fee6a9a370d665db834c961e5d8d30c26273b6ffa71fb"}
Oct 30 02:55:13.778423 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:13.778394    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:55:13.779651 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:13.779607    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-57bd894fd7-n6trx"
Oct 30 02:55:14.093251 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:14.093216    4885 patch_prober.go:29] interesting pod/image-registry-5bbffcc788-vdsgf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 02:55:14.093434 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:14.093282    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf" podUID="bd3e5117-fce3-467f-946b-745eaa13f3e7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 02:55:14.783883 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:14.783834    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4bvl" event={"ID":"5255451b-5afa-4272-b134-38226bfec7d0","Type":"ContainerStarted","Data":"836f92691ceedb3fe36591ddabbfe0011662cb9e78becb6d7ad8aa9508be7e3d"}
Oct 30 02:55:14.785669 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:14.785643    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qggb8" event={"ID":"d921d44f-5ee7-449a-87da-2ebbb52af1aa","Type":"ContainerStarted","Data":"4c5ac12587165f2df3f5e72724b0749e6ac19d07a741a0a4b23bba44ed63d536"}
Oct 30 02:55:14.786064 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:14.786044    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qggb8"
Oct 30 02:55:14.904098 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:14.904063    4885 patch_prober.go:29] interesting pod/image-registry-5bbffcc788-vdsgf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 02:55:14.904256 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:14.904107    4885 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf" podUID="bd3e5117-fce3-467f-946b-745eaa13f3e7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 02:55:16.450977 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:16.450952    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-244kd"
Oct 30 02:55:16.769152 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:16.769120    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x5wjw"
Oct 30 02:55:18.809753 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:18.809723    4885 patch_prober.go:29] interesting pod/image-registry-7cc9c5bc45-q8v5t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 02:55:18.810135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:18.809780    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t" podUID="ed07eb99-f3f5-4dd2-8989-970f992de3fc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 02:55:22.039679 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.039649    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xpgj2"]
Oct 30 02:55:22.084192 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.084163    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.145190 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.145163    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 02:55:22.145325 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.145196    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 02:55:22.167690 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.167658    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebd76df4-8eba-49a7-b254-db08ffb63a04-root\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.167829 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.167694    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-textfile\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.167829 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.167713    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.167829 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.167735    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grxn\" (UniqueName: \"kubernetes.io/projected/ebd76df4-8eba-49a7-b254-db08ffb63a04-kube-api-access-8grxn\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.167952 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.167864    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.167952 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.167903    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-tls\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.167952 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.167930    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-wtmp\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.168061 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.168039    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebd76df4-8eba-49a7-b254-db08ffb63a04-sys\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.168098 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.168080    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebd76df4-8eba-49a7-b254-db08ffb63a04-metrics-client-ca\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.191159 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.191128    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 02:55:22.191159 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.191143    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 02:55:22.191159 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.191128    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4s5rs\""
Oct 30 02:55:22.268963 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.268931    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebd76df4-8eba-49a7-b254-db08ffb63a04-metrics-client-ca\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.268992    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebd76df4-8eba-49a7-b254-db08ffb63a04-root\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269041    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebd76df4-8eba-49a7-b254-db08ffb63a04-root\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269071    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-textfile\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269095    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269123 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269112    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grxn\" (UniqueName: \"kubernetes.io/projected/ebd76df4-8eba-49a7-b254-db08ffb63a04-kube-api-access-8grxn\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269386 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269138    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269386 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269158    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-tls\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269386 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269304    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-wtmp\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269534 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269397    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebd76df4-8eba-49a7-b254-db08ffb63a04-sys\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269534 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269466    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-textfile\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269534 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269479    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-wtmp\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269534 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269496    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebd76df4-8eba-49a7-b254-db08ffb63a04-sys\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269765 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269750    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.269823 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.269802    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebd76df4-8eba-49a7-b254-db08ffb63a04-metrics-client-ca\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.271893 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.271873    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.272005 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.271917    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebd76df4-8eba-49a7-b254-db08ffb63a04-node-exporter-tls\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.290545 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.290489    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grxn\" (UniqueName: \"kubernetes.io/projected/ebd76df4-8eba-49a7-b254-db08ffb63a04-kube-api-access-8grxn\") pod \"node-exporter-xpgj2\" (UID: \"ebd76df4-8eba-49a7-b254-db08ffb63a04\") " pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.436384 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.436351    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xpgj2"
Oct 30 02:55:22.477130 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:55:22.477092    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd76df4_8eba_49a7_b254_db08ffb63a04.slice/crio-53d908445390e26066ba5292e46769ef65b7b3d889238af2d02c9b0e659b71e2 WatchSource:0}: Error finding container 53d908445390e26066ba5292e46769ef65b7b3d889238af2d02c9b0e659b71e2: Status 404 returned error can't find the container with id 53d908445390e26066ba5292e46769ef65b7b3d889238af2d02c9b0e659b71e2
Oct 30 02:55:22.814925 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.814893    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpgj2" event={"ID":"ebd76df4-8eba-49a7-b254-db08ffb63a04","Type":"ContainerStarted","Data":"53d908445390e26066ba5292e46769ef65b7b3d889238af2d02c9b0e659b71e2"}
Oct 30 02:55:22.871678 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:22.871649    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xg9mx"
Oct 30 02:55:24.093699 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:24.093626    4885 patch_prober.go:29] interesting pod/image-registry-5bbffcc788-vdsgf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 02:55:24.093699 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:24.093671    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf" podUID="bd3e5117-fce3-467f-946b-745eaa13f3e7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 02:55:24.823488 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:24.823453    4885 generic.go:356] "Generic (PLEG): container finished" podID="ebd76df4-8eba-49a7-b254-db08ffb63a04" containerID="8e657e66eaf27fea90298be4e97e215af3cd471cf7ce00b10edb24cecffb1758" exitCode=0
Oct 30 02:55:24.823666 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:24.823493    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpgj2" event={"ID":"ebd76df4-8eba-49a7-b254-db08ffb63a04","Type":"ContainerDied","Data":"8e657e66eaf27fea90298be4e97e215af3cd471cf7ce00b10edb24cecffb1758"}
Oct 30 02:55:24.903135 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:24.903108    4885 patch_prober.go:29] interesting pod/image-registry-5bbffcc788-vdsgf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 02:55:24.903256 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:24.903155    4885 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf" podUID="bd3e5117-fce3-467f-946b-745eaa13f3e7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 02:55:25.796738 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:25.796711    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qggb8"
Oct 30 02:55:25.829871 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:25.829842    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpgj2" event={"ID":"ebd76df4-8eba-49a7-b254-db08ffb63a04","Type":"ContainerStarted","Data":"356cc11f2086aef1a35883e6f475fd6611393237d3b571f39abb2a6abe8632fa"}
Oct 30 02:55:25.830008 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:25.829886    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpgj2" event={"ID":"ebd76df4-8eba-49a7-b254-db08ffb63a04","Type":"ContainerStarted","Data":"2748cb6d8389742344b71503e79e2a7e7151c2dcf2cb5acc5f179674ac7b3d88"}
Oct 30 02:55:25.888635 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:25.883222    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xpgj2" podStartSLOduration=3.456385699 podStartE2EDuration="4.883178786s" totalImagesPullingTime="1.350942335s" totalInitContainerRuntime="75.850752ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:25.882059055 +0000 UTC m=+31.392228078" watchObservedRunningTime="2026-10-30 02:55:25.883178786 +0000 UTC m=+31.393347813"
Oct 30 02:55:25.927173 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:55:25.927140    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 02:55:27.944476 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:27.944449    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bbffcc788-vdsgf"]
Oct 30 02:55:27.948761 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:27.948729    4885 patch_prober.go:29] interesting pod/image-registry-5bbffcc788-vdsgf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 02:55:27.948879 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:27.948780    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf" podUID="bd3e5117-fce3-467f-946b-745eaa13f3e7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 02:55:28.807753 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:28.807729    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:55:33.826156 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:33.826116    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t" podUID="ed07eb99-f3f5-4dd2-8989-970f992de3fc" containerName="registry" containerID="cri-o://7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98" gracePeriod=30
Oct 30 02:55:34.067845 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.067823    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:55:34.171010 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.170940    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-certificates\") pod \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") "
Oct 30 02:55:34.171010 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.170981    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") pod \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") "
Oct 30 02:55:34.171218 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.171023    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-bound-sa-token\") pod \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") "
Oct 30 02:55:34.171218 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.171047    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-image-registry-private-configuration\") pod \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") "
Oct 30 02:55:34.171218 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.171066    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-trusted-ca\") pod \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") "
Oct 30 02:55:34.171218 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.171102    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-kube-api-access-7b2xf\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-kube-api-access-7b2xf\") pod \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") "
Oct 30 02:55:34.171218 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.171159    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-installation-pull-secrets\") pod \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") "
Oct 30 02:55:34.171218 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.171206    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed07eb99-f3f5-4dd2-8989-970f992de3fc-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed07eb99-f3f5-4dd2-8989-970f992de3fc-ca-trust-extracted\") pod \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\" (UID: \"ed07eb99-f3f5-4dd2-8989-970f992de3fc\") "
Oct 30 02:55:34.171526 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.171428    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-certificates" pod "ed07eb99-f3f5-4dd2-8989-970f992de3fc" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:34.172167 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.172122    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-trusted-ca" pod "ed07eb99-f3f5-4dd2-8989-970f992de3fc" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:34.173757 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.173694    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls" pod "ed07eb99-f3f5-4dd2-8989-970f992de3fc" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:34.174084 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.174053    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-image-registry-private-configuration" pod "ed07eb99-f3f5-4dd2-8989-970f992de3fc" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:34.174084 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.174061    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-bound-sa-token" pod "ed07eb99-f3f5-4dd2-8989-970f992de3fc" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:34.174234 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.174211    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-installation-pull-secrets" pod "ed07eb99-f3f5-4dd2-8989-970f992de3fc" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:34.174389 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.174366    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-kube-api-access-7b2xf" pod "ed07eb99-f3f5-4dd2-8989-970f992de3fc" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc"). InnerVolumeSpecName "kube-api-access-7b2xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:34.177878 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.177860    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed07eb99-f3f5-4dd2-8989-970f992de3fc-ca-trust-extracted" pod "ed07eb99-f3f5-4dd2-8989-970f992de3fc" (UID: "ed07eb99-f3f5-4dd2-8989-970f992de3fc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:55:34.272338 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.272312    4885 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed07eb99-f3f5-4dd2-8989-970f992de3fc-ca-trust-extracted\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:34.272338 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.272333    4885 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-certificates\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:34.272491 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.272345    4885 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-registry-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:34.272491 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.272354    4885 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-bound-sa-token\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:34.272491 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.272363    4885 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-image-registry-private-configuration\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:34.272491 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.272371    4885 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed07eb99-f3f5-4dd2-8989-970f992de3fc-trusted-ca\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:34.272491 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.272379    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b2xf\" (UniqueName: \"kubernetes.io/projected/ed07eb99-f3f5-4dd2-8989-970f992de3fc-kube-api-access-7b2xf\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:34.272491 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.272387    4885 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed07eb99-f3f5-4dd2-8989-970f992de3fc-installation-pull-secrets\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:34.864018 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.863987    4885 generic.go:356] "Generic (PLEG): container finished" podID="ed07eb99-f3f5-4dd2-8989-970f992de3fc" containerID="7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98" exitCode=0
Oct 30 02:55:34.864416 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.864042    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"
Oct 30 02:55:34.864416 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.864058    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t" event={"ID":"ed07eb99-f3f5-4dd2-8989-970f992de3fc","Type":"ContainerDied","Data":"7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98"}
Oct 30 02:55:34.864416 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.864144    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cc9c5bc45-q8v5t" event={"ID":"ed07eb99-f3f5-4dd2-8989-970f992de3fc","Type":"ContainerDied","Data":"235fa1b42555a5ef39e9a2ffaa755472310b94d6d526ec1fc281cd479c18a648"}
Oct 30 02:55:34.864416 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.864171    4885 scope.go:122] "RemoveContainer" containerID="7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98"
Oct 30 02:55:34.872427 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.872406    4885 scope.go:122] "RemoveContainer" containerID="7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98"
Oct 30 02:55:34.872696 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:55:34.872678    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98\": container with ID starting with 7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98 not found: ID does not exist" containerID="7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98"
Oct 30 02:55:34.872754 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.872708    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98"} err="failed to get container status \"7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98\": rpc error: code = NotFound desc = could not find container \"7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98\": container with ID starting with 7385a879ea79d0bd21b19f00e1b4accb4d3b79ffabd20166a04151ffe2d6ae98 not found: ID does not exist"
Oct 30 02:55:34.919151 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.919095    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"]
Oct 30 02:55:34.929391 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:34.929366    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cc9c5bc45-q8v5t"]
Oct 30 02:55:36.623488 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:36.623414    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed07eb99-f3f5-4dd2-8989-970f992de3fc" path="/var/lib/kubelet/pods/ed07eb99-f3f5-4dd2-8989-970f992de3fc/volumes"
Oct 30 02:55:37.949027 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:37.949004    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:55:52.964677 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:52.964094    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf" podUID="bd3e5117-fce3-467f-946b-745eaa13f3e7" containerName="registry" containerID="cri-o://f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c" gracePeriod=30
Oct 30 02:55:53.210923 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.210899    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:55:53.327246 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.327172    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-kube-api-access-29fjt\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-kube-api-access-29fjt\") pod \"bd3e5117-fce3-467f-946b-745eaa13f3e7\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") "
Oct 30 02:55:53.327246 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.327209    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-certificates\") pod \"bd3e5117-fce3-467f-946b-745eaa13f3e7\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") "
Oct 30 02:55:53.327246 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.327239    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") pod \"bd3e5117-fce3-467f-946b-745eaa13f3e7\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") "
Oct 30 02:55:53.327445 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.327256    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-trusted-ca\") pod \"bd3e5117-fce3-467f-946b-745eaa13f3e7\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") "
Oct 30 02:55:53.327478 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.327436    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-bound-sa-token\") pod \"bd3e5117-fce3-467f-946b-745eaa13f3e7\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") "
Oct 30 02:55:53.327543 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.327526    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-installation-pull-secrets\") pod \"bd3e5117-fce3-467f-946b-745eaa13f3e7\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") "
Oct 30 02:55:53.327600 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.327584    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3e5117-fce3-467f-946b-745eaa13f3e7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd3e5117-fce3-467f-946b-745eaa13f3e7-ca-trust-extracted\") pod \"bd3e5117-fce3-467f-946b-745eaa13f3e7\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") "
Oct 30 02:55:53.327694 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.327657    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-certificates" pod "bd3e5117-fce3-467f-946b-745eaa13f3e7" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:53.327694 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.327668    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-trusted-ca" pod "bd3e5117-fce3-467f-946b-745eaa13f3e7" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:53.327694 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.327662    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-image-registry-private-configuration\") pod \"bd3e5117-fce3-467f-946b-745eaa13f3e7\" (UID: \"bd3e5117-fce3-467f-946b-745eaa13f3e7\") "
Oct 30 02:55:53.328078 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.328054    4885 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-certificates\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:53.328078 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.328078    4885 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e5117-fce3-467f-946b-745eaa13f3e7-trusted-ca\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:53.330310 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.330279    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls" pod "bd3e5117-fce3-467f-946b-745eaa13f3e7" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:53.330310 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.330285    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-kube-api-access-29fjt" pod "bd3e5117-fce3-467f-946b-745eaa13f3e7" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7"). InnerVolumeSpecName "kube-api-access-29fjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:53.330437 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.330311    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-image-registry-private-configuration" pod "bd3e5117-fce3-467f-946b-745eaa13f3e7" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:53.330437 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.330394    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-bound-sa-token" pod "bd3e5117-fce3-467f-946b-745eaa13f3e7" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:53.331784 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.331764    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-installation-pull-secrets" pod "bd3e5117-fce3-467f-946b-745eaa13f3e7" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:53.335061 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.335033    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3e5117-fce3-467f-946b-745eaa13f3e7-ca-trust-extracted" pod "bd3e5117-fce3-467f-946b-745eaa13f3e7" (UID: "bd3e5117-fce3-467f-946b-745eaa13f3e7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:55:53.429388 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.429358    4885 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-image-registry-private-configuration\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:53.429388 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.429385    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29fjt\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-kube-api-access-29fjt\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:53.429388 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.429396    4885 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-registry-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:53.429697 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.429405    4885 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd3e5117-fce3-467f-946b-745eaa13f3e7-bound-sa-token\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:53.429697 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.429413    4885 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd3e5117-fce3-467f-946b-745eaa13f3e7-installation-pull-secrets\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:53.429697 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.429422    4885 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd3e5117-fce3-467f-946b-745eaa13f3e7-ca-trust-extracted\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:55:53.936946 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.936908    4885 generic.go:356] "Generic (PLEG): container finished" podID="bd3e5117-fce3-467f-946b-745eaa13f3e7" containerID="f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c" exitCode=0
Oct 30 02:55:53.937114 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.936956    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf" event={"ID":"bd3e5117-fce3-467f-946b-745eaa13f3e7","Type":"ContainerDied","Data":"f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c"}
Oct 30 02:55:53.937114 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.936975    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf"
Oct 30 02:55:53.937114 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.937005    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bbffcc788-vdsgf" event={"ID":"bd3e5117-fce3-467f-946b-745eaa13f3e7","Type":"ContainerDied","Data":"2b3ca2e43186722dfcc108b1452810c416fc44390af680ebab500f6f439fd057"}
Oct 30 02:55:53.937114 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.937033    4885 scope.go:122] "RemoveContainer" containerID="f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c"
Oct 30 02:55:53.945447 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.945430    4885 scope.go:122] "RemoveContainer" containerID="f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c"
Oct 30 02:55:53.945738 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:55:53.945717    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c\": container with ID starting with f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c not found: ID does not exist" containerID="f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c"
Oct 30 02:55:53.945787 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.945746    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c"} err="failed to get container status \"f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c\": rpc error: code = NotFound desc = could not find container \"f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c\": container with ID starting with f57b3ab75a0ca1e23805161712c19b7236fbb4426aadb05a7b83eddabcf7be5c not found: ID does not exist"
Oct 30 02:55:53.992903 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:53.992875    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bbffcc788-vdsgf"]
Oct 30 02:55:54.003131 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:54.003109    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bbffcc788-vdsgf"]
Oct 30 02:55:54.622505 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:55:54.622475    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3e5117-fce3-467f-946b-745eaa13f3e7" path="/var/lib/kubelet/pods/bd3e5117-fce3-467f-946b-745eaa13f3e7/volumes"
Oct 30 02:55:55.921744 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:55:55.921707    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 02:56:08.990982 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:08.990951    4885 generic.go:356] "Generic (PLEG): container finished" podID="2e4c1ab5-989e-4fe6-a028-85579a18a996" containerID="9bb8d2d71354a5a99af612879718e4dd66ae73f37a22e9a17dcc05f6472523aa" exitCode=0
Oct 30 02:56:08.991400 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:08.991025    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7zx8h" event={"ID":"2e4c1ab5-989e-4fe6-a028-85579a18a996","Type":"ContainerDied","Data":"9bb8d2d71354a5a99af612879718e4dd66ae73f37a22e9a17dcc05f6472523aa"}
Oct 30 02:56:08.991400 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:08.991329    4885 scope.go:122] "RemoveContainer" containerID="9bb8d2d71354a5a99af612879718e4dd66ae73f37a22e9a17dcc05f6472523aa"
Oct 30 02:56:09.747266 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.747236    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-8ld9l-8zq7p"]
Oct 30 02:56:09.747657 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.747638    4885 cpu_manager.go:414] "containerMap: removing container" podUID="ed07eb99-f3f5-4dd2-8989-970f992de3fc" containerName="registry"
Oct 30 02:56:09.747735 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.747661    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed07eb99-f3f5-4dd2-8989-970f992de3fc" containerName="registry"
Oct 30 02:56:09.748237 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.748219    4885 cpu_manager.go:414] "containerMap: removing container" podUID="bd3e5117-fce3-467f-946b-745eaa13f3e7" containerName="registry"
Oct 30 02:56:09.748237 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.748236    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3e5117-fce3-467f-946b-745eaa13f3e7" containerName="registry"
Oct 30 02:56:09.748345 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.748319    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed07eb99-f3f5-4dd2-8989-970f992de3fc" containerName="registry"
Oct 30 02:56:09.748345 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.748330    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3e5117-fce3-467f-946b-745eaa13f3e7" containerName="registry"
Oct 30 02:56:09.751148 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.751133    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.773967 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.773941    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-l5dqk\""
Oct 30 02:56:09.775304 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.775285    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8ld9l-8zq7p"]
Oct 30 02:56:09.855707 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.855676    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-trusted-ca-bundle\") pod \"periodic-gathering-8ld9l-8zq7p\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") " pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.855873 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.855720    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khbn9\" (UniqueName: \"kubernetes.io/projected/6d93c572-ccd9-4cd0-a604-72b6e33aae57-kube-api-access-khbn9\") pod \"periodic-gathering-8ld9l-8zq7p\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") " pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.855873 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.855744    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-service-ca-bundle\") pod \"periodic-gathering-8ld9l-8zq7p\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") " pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.855985 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.855870    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6d93c572-ccd9-4cd0-a604-72b6e33aae57-archives-path\") pod \"periodic-gathering-8ld9l-8zq7p\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") " pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.957059 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.957026    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khbn9\" (UniqueName: \"kubernetes.io/projected/6d93c572-ccd9-4cd0-a604-72b6e33aae57-kube-api-access-khbn9\") pod \"periodic-gathering-8ld9l-8zq7p\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") " pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.957253 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.957075    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-service-ca-bundle\") pod \"periodic-gathering-8ld9l-8zq7p\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") " pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.957253 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.957119    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6d93c572-ccd9-4cd0-a604-72b6e33aae57-archives-path\") pod \"periodic-gathering-8ld9l-8zq7p\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") " pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.957253 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.957171    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-trusted-ca-bundle\") pod \"periodic-gathering-8ld9l-8zq7p\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") " pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.957697 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.957669    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6d93c572-ccd9-4cd0-a604-72b6e33aae57-archives-path\") pod \"periodic-gathering-8ld9l-8zq7p\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") " pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.958085 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.958058    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-service-ca-bundle\") pod \"periodic-gathering-8ld9l-8zq7p\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") " pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.958345 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.958323    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-trusted-ca-bundle\") pod \"periodic-gathering-8ld9l-8zq7p\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") " pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.979140 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.979109    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khbn9\" (UniqueName: \"kubernetes.io/projected/6d93c572-ccd9-4cd0-a604-72b6e33aae57-kube-api-access-khbn9\") pod \"periodic-gathering-8ld9l-8zq7p\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") " pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:09.996605 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:09.996578    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7zx8h" event={"ID":"2e4c1ab5-989e-4fe6-a028-85579a18a996","Type":"ContainerStarted","Data":"7daf5c54c346415599e274eb01b18c5876af69e60d4f33dada21429b2c70ed9c"}
Oct 30 02:56:10.062668 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:10.062579    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:56:10.355771 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:10.355681    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8ld9l-8zq7p"]
Oct 30 02:56:10.358485 ip-10-0-140-132 kubenswrapper[4885]: W1030 02:56:10.358455    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d93c572_ccd9_4cd0_a604_72b6e33aae57.slice/crio-a150665f0bd2eadbea63c334ce0990f912aa03f33cdc9eb6f1799ad065d9fddf WatchSource:0}: Error finding container a150665f0bd2eadbea63c334ce0990f912aa03f33cdc9eb6f1799ad065d9fddf: Status 404 returned error can't find the container with id a150665f0bd2eadbea63c334ce0990f912aa03f33cdc9eb6f1799ad065d9fddf
Oct 30 02:56:11.001329 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:11.001292    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8ld9l-8zq7p" event={"ID":"6d93c572-ccd9-4cd0-a604-72b6e33aae57","Type":"ContainerStarted","Data":"c0117a3079bb81c40b367406ec854b2f50c813733b0411da39c89e82462e8101"}
Oct 30 02:56:11.001751 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:11.001343    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8ld9l-8zq7p" event={"ID":"6d93c572-ccd9-4cd0-a604-72b6e33aae57","Type":"ContainerStarted","Data":"a150665f0bd2eadbea63c334ce0990f912aa03f33cdc9eb6f1799ad065d9fddf"}
Oct 30 02:56:11.038391 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:11.038341    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-8ld9l-8zq7p" podStartSLOduration=2.038328153 podStartE2EDuration="2.038328153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:11.037852838 +0000 UTC m=+76.548021887" watchObservedRunningTime="2026-10-30 02:56:11.038328153 +0000 UTC m=+76.548497179"
Oct 30 02:56:15.021174 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:15.021138    4885 generic.go:356] "Generic (PLEG): container finished" podID="dbb2d265-64b9-4e4f-be5b-1f00ba684673" containerID="5d255ab3a482cac9e79c367f1b8384971aee14f3937ef6d83e24cb17bb6b0f96" exitCode=0
Oct 30 02:56:15.021560 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:15.021199    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8" event={"ID":"dbb2d265-64b9-4e4f-be5b-1f00ba684673","Type":"ContainerDied","Data":"5d255ab3a482cac9e79c367f1b8384971aee14f3937ef6d83e24cb17bb6b0f96"}
Oct 30 02:56:15.021560 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:15.021514    4885 scope.go:122] "RemoveContainer" containerID="5d255ab3a482cac9e79c367f1b8384971aee14f3937ef6d83e24cb17bb6b0f96"
Oct 30 02:56:16.026991 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:56:16.026954    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4pbx8" event={"ID":"dbb2d265-64b9-4e4f-be5b-1f00ba684673","Type":"ContainerStarted","Data":"da5781696d259a63fc9b2c5d762a03a7c0ee0ffd58222f744e8ba398bd21ab4f"}
Oct 30 02:56:25.921768 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:56:25.921731    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 02:56:55.921574 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:56:55.921527    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 02:57:25.922638 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:57:25.922544    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 02:57:34.307576 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:34.307545    4885 generic.go:356] "Generic (PLEG): container finished" podID="6d93c572-ccd9-4cd0-a604-72b6e33aae57" containerID="c0117a3079bb81c40b367406ec854b2f50c813733b0411da39c89e82462e8101" exitCode=0
Oct 30 02:57:34.307983 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:34.307631    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8ld9l-8zq7p" event={"ID":"6d93c572-ccd9-4cd0-a604-72b6e33aae57","Type":"ContainerDied","Data":"c0117a3079bb81c40b367406ec854b2f50c813733b0411da39c89e82462e8101"}
Oct 30 02:57:35.438776 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.438753    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:57:35.525403 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.525370    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d93c572-ccd9-4cd0-a604-72b6e33aae57-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6d93c572-ccd9-4cd0-a604-72b6e33aae57-archives-path\") pod \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") "
Oct 30 02:57:35.525555 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.525420    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d93c572-ccd9-4cd0-a604-72b6e33aae57-kube-api-access-khbn9\" (UniqueName: \"kubernetes.io/projected/6d93c572-ccd9-4cd0-a604-72b6e33aae57-kube-api-access-khbn9\") pod \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") "
Oct 30 02:57:35.525555 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.525441    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-trusted-ca-bundle\") pod \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") "
Oct 30 02:57:35.525744 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.525722    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d93c572-ccd9-4cd0-a604-72b6e33aae57-archives-path" pod "6d93c572-ccd9-4cd0-a604-72b6e33aae57" (UID: "6d93c572-ccd9-4cd0-a604-72b6e33aae57"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:57:35.525843 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.525826    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-trusted-ca-bundle" pod "6d93c572-ccd9-4cd0-a604-72b6e33aae57" (UID: "6d93c572-ccd9-4cd0-a604-72b6e33aae57"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:35.527796 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.527770    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d93c572-ccd9-4cd0-a604-72b6e33aae57-kube-api-access-khbn9" pod "6d93c572-ccd9-4cd0-a604-72b6e33aae57" (UID: "6d93c572-ccd9-4cd0-a604-72b6e33aae57"). InnerVolumeSpecName "kube-api-access-khbn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:35.626075 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.625990    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-service-ca-bundle\") pod \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\" (UID: \"6d93c572-ccd9-4cd0-a604-72b6e33aae57\") "
Oct 30 02:57:35.626251 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.626230    4885 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6d93c572-ccd9-4cd0-a604-72b6e33aae57-archives-path\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:57:35.626251 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.626241    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-service-ca-bundle" pod "6d93c572-ccd9-4cd0-a604-72b6e33aae57" (UID: "6d93c572-ccd9-4cd0-a604-72b6e33aae57"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:35.626356 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.626328    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khbn9\" (UniqueName: \"kubernetes.io/projected/6d93c572-ccd9-4cd0-a604-72b6e33aae57-kube-api-access-khbn9\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:57:35.626356 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.626347    4885 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-trusted-ca-bundle\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:57:35.726926 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:35.726901    4885 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d93c572-ccd9-4cd0-a604-72b6e33aae57-service-ca-bundle\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 02:57:36.315148 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:36.315118    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8ld9l-8zq7p"
Oct 30 02:57:36.315316 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:36.315112    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8ld9l-8zq7p" event={"ID":"6d93c572-ccd9-4cd0-a604-72b6e33aae57","Type":"ContainerDied","Data":"a150665f0bd2eadbea63c334ce0990f912aa03f33cdc9eb6f1799ad065d9fddf"}
Oct 30 02:57:36.315316 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:57:36.315232    4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a150665f0bd2eadbea63c334ce0990f912aa03f33cdc9eb6f1799ad065d9fddf"
Oct 30 02:57:55.922643 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:57:55.922590    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 02:58:25.921070 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:58:25.921030    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 02:58:55.920921 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:58:55.920877    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 02:59:25.921734 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:59:25.921687    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 02:59:54.639469 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:59:54.639443    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 02:59:54.639966 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:59:54.639443    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 02:59:54.644441 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:59:54.644424    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 02:59:54.644778 ip-10-0-140-132 kubenswrapper[4885]: I1030 02:59:54.644761    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 02:59:55.921099 ip-10-0-140-132 kubenswrapper[4885]: E1030 02:59:55.921057    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:00:25.921399 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:00:25.921310    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:00:55.923890 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:00:55.923844    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:01:25.921286 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:01:25.921251    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:01:55.921763 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:01:55.921724    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:02:25.922038 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:02:25.922000    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:02:55.926321 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:02:55.926266    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:03:25.921738 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:03:25.921661    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:03:26.715305 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.715275    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-d55b668ff-l4s4n"]
Oct 30 03:03:26.715573 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.715562    4885 cpu_manager.go:414] "containerMap: removing container" podUID="6d93c572-ccd9-4cd0-a604-72b6e33aae57" containerName="insights-gathering"
Oct 30 03:03:26.715633 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.715575    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d93c572-ccd9-4cd0-a604-72b6e33aae57" containerName="insights-gathering"
Oct 30 03:03:26.715669 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.715653    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d93c572-ccd9-4cd0-a604-72b6e33aae57" containerName="insights-gathering"
Oct 30 03:03:26.718572 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.718551    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-d55b668ff-l4s4n"
Oct 30 03:03:26.741400 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.741371    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-5c27k\""
Oct 30 03:03:26.741400 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.741372    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Oct 30 03:03:26.741698 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.741679    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 03:03:26.742691 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.742633    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-d55b668ff-l4s4n"]
Oct 30 03:03:26.744601 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.744580    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 03:03:26.790696 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.790671    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-pzrnh"]
Oct 30 03:03:26.794008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.793990    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-pzrnh"
Oct 30 03:03:26.814397 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.814374    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-pzrnh"]
Oct 30 03:03:26.834812 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.834786    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 30 03:03:26.834941 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.834927    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-55fdx\""
Oct 30 03:03:26.898405 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.898372    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztkq7\" (UniqueName: \"kubernetes.io/projected/d4c60f6e-7fb0-4e53-af8c-6209c78599c8-kube-api-access-ztkq7\") pod \"seaweedfs-78bb9c6ff8-pzrnh\" (UID: \"d4c60f6e-7fb0-4e53-af8c-6209c78599c8\") " pod="kserve/seaweedfs-78bb9c6ff8-pzrnh"
Oct 30 03:03:26.898569 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.898411    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d4c60f6e-7fb0-4e53-af8c-6209c78599c8-data\") pod \"seaweedfs-78bb9c6ff8-pzrnh\" (UID: \"d4c60f6e-7fb0-4e53-af8c-6209c78599c8\") " pod="kserve/seaweedfs-78bb9c6ff8-pzrnh"
Oct 30 03:03:26.898569 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.898552    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-cert\") pod \"kserve-controller-manager-d55b668ff-l4s4n\" (UID: \"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643\") " pod="kserve/kserve-controller-manager-d55b668ff-l4s4n"
Oct 30 03:03:26.898699 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.898587    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65bl\" (UniqueName: \"kubernetes.io/projected/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-kube-api-access-x65bl\") pod \"kserve-controller-manager-d55b668ff-l4s4n\" (UID: \"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643\") " pod="kserve/kserve-controller-manager-d55b668ff-l4s4n"
Oct 30 03:03:26.999921 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.999826    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-cert\") pod \"kserve-controller-manager-d55b668ff-l4s4n\" (UID: \"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643\") " pod="kserve/kserve-controller-manager-d55b668ff-l4s4n"
Oct 30 03:03:26.999921 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.999875    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65bl\" (UniqueName: \"kubernetes.io/projected/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-kube-api-access-x65bl\") pod \"kserve-controller-manager-d55b668ff-l4s4n\" (UID: \"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643\") " pod="kserve/kserve-controller-manager-d55b668ff-l4s4n"
Oct 30 03:03:26.999921 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.999919    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztkq7\" (UniqueName: \"kubernetes.io/projected/d4c60f6e-7fb0-4e53-af8c-6209c78599c8-kube-api-access-ztkq7\") pod \"seaweedfs-78bb9c6ff8-pzrnh\" (UID: \"d4c60f6e-7fb0-4e53-af8c-6209c78599c8\") " pod="kserve/seaweedfs-78bb9c6ff8-pzrnh"
Oct 30 03:03:27.000461 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:26.999935    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d4c60f6e-7fb0-4e53-af8c-6209c78599c8-data\") pod \"seaweedfs-78bb9c6ff8-pzrnh\" (UID: \"d4c60f6e-7fb0-4e53-af8c-6209c78599c8\") " pod="kserve/seaweedfs-78bb9c6ff8-pzrnh"
Oct 30 03:03:27.000561 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:27.000544    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d4c60f6e-7fb0-4e53-af8c-6209c78599c8-data\") pod \"seaweedfs-78bb9c6ff8-pzrnh\" (UID: \"d4c60f6e-7fb0-4e53-af8c-6209c78599c8\") " pod="kserve/seaweedfs-78bb9c6ff8-pzrnh"
Oct 30 03:03:27.002808 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:27.002786    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-cert\") pod \"kserve-controller-manager-d55b668ff-l4s4n\" (UID: \"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643\") " pod="kserve/kserve-controller-manager-d55b668ff-l4s4n"
Oct 30 03:03:27.022745 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:27.022717    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65bl\" (UniqueName: \"kubernetes.io/projected/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-kube-api-access-x65bl\") pod \"kserve-controller-manager-d55b668ff-l4s4n\" (UID: \"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643\") " pod="kserve/kserve-controller-manager-d55b668ff-l4s4n"
Oct 30 03:03:27.022835 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:27.022787    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztkq7\" (UniqueName: \"kubernetes.io/projected/d4c60f6e-7fb0-4e53-af8c-6209c78599c8-kube-api-access-ztkq7\") pod \"seaweedfs-78bb9c6ff8-pzrnh\" (UID: \"d4c60f6e-7fb0-4e53-af8c-6209c78599c8\") " pod="kserve/seaweedfs-78bb9c6ff8-pzrnh"
Oct 30 03:03:27.030577 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:27.030558    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-d55b668ff-l4s4n"
Oct 30 03:03:27.104722 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:27.104694    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-pzrnh"
Oct 30 03:03:27.321501 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:27.321294    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-d55b668ff-l4s4n"]
Oct 30 03:03:27.324119 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:03:27.324095    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf6c76a_cec0_49c7_bb38_cbfa84fa0643.slice/crio-d0c8a60748075fb13e59a47c055bf57da53c0667e8c5cc2cea69f3d82507a8a5 WatchSource:0}: Error finding container d0c8a60748075fb13e59a47c055bf57da53c0667e8c5cc2cea69f3d82507a8a5: Status 404 returned error can't find the container with id d0c8a60748075fb13e59a47c055bf57da53c0667e8c5cc2cea69f3d82507a8a5
Oct 30 03:03:27.325358 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:27.325341    4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:03:27.393391 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:27.393361    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-pzrnh"]
Oct 30 03:03:27.396397 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:03:27.396355    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c60f6e_7fb0_4e53_af8c_6209c78599c8.slice/crio-11268a7d7b39a4cbd4163ad188e1de119866481a4fd05a4c89eecb939ac26010 WatchSource:0}: Error finding container 11268a7d7b39a4cbd4163ad188e1de119866481a4fd05a4c89eecb939ac26010: Status 404 returned error can't find the container with id 11268a7d7b39a4cbd4163ad188e1de119866481a4fd05a4c89eecb939ac26010
Oct 30 03:03:27.533326 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:27.533293    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-d55b668ff-l4s4n" event={"ID":"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643","Type":"ContainerStarted","Data":"d0c8a60748075fb13e59a47c055bf57da53c0667e8c5cc2cea69f3d82507a8a5"}
Oct 30 03:03:27.534322 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:27.534290    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-pzrnh" event={"ID":"d4c60f6e-7fb0-4e53-af8c-6209c78599c8","Type":"ContainerStarted","Data":"11268a7d7b39a4cbd4163ad188e1de119866481a4fd05a4c89eecb939ac26010"}
Oct 30 03:03:30.561038 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:30.561008    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-d55b668ff-l4s4n" event={"ID":"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643","Type":"ContainerStarted","Data":"c27806274aeac24f41e0ce816fc7ba90e67f5794e7b271dd939a63f0a9ac4a1f"}
Oct 30 03:03:30.561489 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:30.561077    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-d55b668ff-l4s4n"
Oct 30 03:03:30.598654 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:30.598582    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-d55b668ff-l4s4n" podStartSLOduration=1.752062478 podStartE2EDuration="4.598566665s" totalImagesPullingTime="2.846504187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:30.597787359 +0000 UTC m=+516.107956402" watchObservedRunningTime="2026-10-30 03:03:30.598566665 +0000 UTC m=+516.108735732"
Oct 30 03:03:31.566774 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:31.566740    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-pzrnh" event={"ID":"d4c60f6e-7fb0-4e53-af8c-6209c78599c8","Type":"ContainerStarted","Data":"b0d7ef20e19d676a77c6927b81567067dc6228b5255f101aeab865686f77f7c9"}
Oct 30 03:03:31.567279 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:31.566933    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-78bb9c6ff8-pzrnh"
Oct 30 03:03:31.603052 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:31.603004    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-78bb9c6ff8-pzrnh" podStartSLOduration=2.299420035 podStartE2EDuration="5.602988672s" totalImagesPullingTime="3.303568637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:31.602404439 +0000 UTC m=+517.112573466" watchObservedRunningTime="2026-10-30 03:03:31.602988672 +0000 UTC m=+517.113157707"
Oct 30 03:03:37.572840 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:03:37.572807    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-78bb9c6ff8-pzrnh"
Oct 30 03:03:55.920719 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:03:55.920684    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:04:01.572933 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:01.572902    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-d55b668ff-l4s4n"
Oct 30 03:04:15.505205 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.505173    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-d55b668ff-l4s4n"]
Oct 30 03:04:15.505765 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.505370    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="kserve/kserve-controller-manager-d55b668ff-l4s4n" podUID="7bf6c76a-cec0-49c7-bb38-cbfa84fa0643" containerName="manager" containerID="cri-o://c27806274aeac24f41e0ce816fc7ba90e67f5794e7b271dd939a63f0a9ac4a1f" gracePeriod=10
Oct 30 03:04:15.531697 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.531671    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-d55b668ff-msg2d"]
Oct 30 03:04:15.533756 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.533741    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-d55b668ff-msg2d"
Oct 30 03:04:15.555138 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.555110    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-d55b668ff-msg2d"]
Oct 30 03:04:15.656206 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.656168    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv5gn\" (UniqueName: \"kubernetes.io/projected/675d2de2-30b9-48d6-8a52-d200e3a54ad5-kube-api-access-kv5gn\") pod \"kserve-controller-manager-d55b668ff-msg2d\" (UID: \"675d2de2-30b9-48d6-8a52-d200e3a54ad5\") " pod="kserve/kserve-controller-manager-d55b668ff-msg2d"
Oct 30 03:04:15.656362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.656322    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/675d2de2-30b9-48d6-8a52-d200e3a54ad5-cert\") pod \"kserve-controller-manager-d55b668ff-msg2d\" (UID: \"675d2de2-30b9-48d6-8a52-d200e3a54ad5\") " pod="kserve/kserve-controller-manager-d55b668ff-msg2d"
Oct 30 03:04:15.736071 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.736042    4885 generic.go:356] "Generic (PLEG): container finished" podID="7bf6c76a-cec0-49c7-bb38-cbfa84fa0643" containerID="c27806274aeac24f41e0ce816fc7ba90e67f5794e7b271dd939a63f0a9ac4a1f" exitCode=0
Oct 30 03:04:15.736187 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.736070    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-d55b668ff-l4s4n" event={"ID":"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643","Type":"ContainerDied","Data":"c27806274aeac24f41e0ce816fc7ba90e67f5794e7b271dd939a63f0a9ac4a1f"}
Oct 30 03:04:15.736187 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.736121    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-d55b668ff-l4s4n" event={"ID":"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643","Type":"ContainerDied","Data":"d0c8a60748075fb13e59a47c055bf57da53c0667e8c5cc2cea69f3d82507a8a5"}
Oct 30 03:04:15.736187 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.736143    4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0c8a60748075fb13e59a47c055bf57da53c0667e8c5cc2cea69f3d82507a8a5"
Oct 30 03:04:15.744261 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.744242    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-d55b668ff-l4s4n"
Oct 30 03:04:15.757212 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.757156    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv5gn\" (UniqueName: \"kubernetes.io/projected/675d2de2-30b9-48d6-8a52-d200e3a54ad5-kube-api-access-kv5gn\") pod \"kserve-controller-manager-d55b668ff-msg2d\" (UID: \"675d2de2-30b9-48d6-8a52-d200e3a54ad5\") " pod="kserve/kserve-controller-manager-d55b668ff-msg2d"
Oct 30 03:04:15.757308 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.757225    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/675d2de2-30b9-48d6-8a52-d200e3a54ad5-cert\") pod \"kserve-controller-manager-d55b668ff-msg2d\" (UID: \"675d2de2-30b9-48d6-8a52-d200e3a54ad5\") " pod="kserve/kserve-controller-manager-d55b668ff-msg2d"
Oct 30 03:04:15.759452 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.759430    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/675d2de2-30b9-48d6-8a52-d200e3a54ad5-cert\") pod \"kserve-controller-manager-d55b668ff-msg2d\" (UID: \"675d2de2-30b9-48d6-8a52-d200e3a54ad5\") " pod="kserve/kserve-controller-manager-d55b668ff-msg2d"
Oct 30 03:04:15.778768 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.778747    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv5gn\" (UniqueName: \"kubernetes.io/projected/675d2de2-30b9-48d6-8a52-d200e3a54ad5-kube-api-access-kv5gn\") pod \"kserve-controller-manager-d55b668ff-msg2d\" (UID: \"675d2de2-30b9-48d6-8a52-d200e3a54ad5\") " pod="kserve/kserve-controller-manager-d55b668ff-msg2d"
Oct 30 03:04:15.858365 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.858331    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-cert\" (UniqueName: \"kubernetes.io/secret/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-cert\") pod \"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643\" (UID: \"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643\") "
Oct 30 03:04:15.858517 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.858374    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-kube-api-access-x65bl\" (UniqueName: \"kubernetes.io/projected/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-kube-api-access-x65bl\") pod \"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643\" (UID: \"7bf6c76a-cec0-49c7-bb38-cbfa84fa0643\") "
Oct 30 03:04:15.860651 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.860604    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-kube-api-access-x65bl" pod "7bf6c76a-cec0-49c7-bb38-cbfa84fa0643" (UID: "7bf6c76a-cec0-49c7-bb38-cbfa84fa0643"). InnerVolumeSpecName "kube-api-access-x65bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:15.860747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.860663    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-cert" pod "7bf6c76a-cec0-49c7-bb38-cbfa84fa0643" (UID: "7bf6c76a-cec0-49c7-bb38-cbfa84fa0643"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:04:15.889816 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.889792    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-d55b668ff-msg2d"
Oct 30 03:04:15.959929 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.959906    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x65bl\" (UniqueName: \"kubernetes.io/projected/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-kube-api-access-x65bl\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:04:15.959929 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:15.959928    4885 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:04:16.178691 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:16.178591    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-d55b668ff-msg2d"]
Oct 30 03:04:16.181260 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:04:16.181234    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675d2de2_30b9_48d6_8a52_d200e3a54ad5.slice/crio-d7aa574480618ff738a1ac3d0feac41a18ff5b28c24b8cda6e71d2a3eec22ccb WatchSource:0}: Error finding container d7aa574480618ff738a1ac3d0feac41a18ff5b28c24b8cda6e71d2a3eec22ccb: Status 404 returned error can't find the container with id d7aa574480618ff738a1ac3d0feac41a18ff5b28c24b8cda6e71d2a3eec22ccb
Oct 30 03:04:16.742215 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:16.742186    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-d55b668ff-msg2d" event={"ID":"675d2de2-30b9-48d6-8a52-d200e3a54ad5","Type":"ContainerStarted","Data":"79a96aacfc755f63371f459561283f21b935fe674c549c352fc6bb136f8310fd"}
Oct 30 03:04:16.742641 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:16.742234    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-d55b668ff-msg2d" event={"ID":"675d2de2-30b9-48d6-8a52-d200e3a54ad5","Type":"ContainerStarted","Data":"d7aa574480618ff738a1ac3d0feac41a18ff5b28c24b8cda6e71d2a3eec22ccb"}
Oct 30 03:04:16.742641 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:16.742198    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-d55b668ff-l4s4n"
Oct 30 03:04:16.742641 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:16.742287    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-d55b668ff-msg2d"
Oct 30 03:04:16.779840 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:16.779745    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-d55b668ff-msg2d" podStartSLOduration=1.413116337 podStartE2EDuration="1.779732586s" totalImagesPullingTime="366.616249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:16.779668139 +0000 UTC m=+562.289837183" watchObservedRunningTime="2026-10-30 03:04:16.779732586 +0000 UTC m=+562.289901611"
Oct 30 03:04:16.815750 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:16.815729    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-d55b668ff-l4s4n"]
Oct 30 03:04:16.818241 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:16.818222    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-d55b668ff-l4s4n"]
Oct 30 03:04:18.624434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:18.624400    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf6c76a-cec0-49c7-bb38-cbfa84fa0643" path="/var/lib/kubelet/pods/7bf6c76a-cec0-49c7-bb38-cbfa84fa0643/volumes"
Oct 30 03:04:25.921372 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:04:25.921327    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:04:47.754203 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:47.754175    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-d55b668ff-msg2d"
Oct 30 03:04:50.605660 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.605629    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-56ddc4c448-vx8p5"]
Oct 30 03:04:50.606065 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.605932    4885 cpu_manager.go:414] "containerMap: removing container" podUID="7bf6c76a-cec0-49c7-bb38-cbfa84fa0643" containerName="manager"
Oct 30 03:04:50.606065 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.605943    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf6c76a-cec0-49c7-bb38-cbfa84fa0643" containerName="manager"
Oct 30 03:04:50.606065 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.605996    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf6c76a-cec0-49c7-bb38-cbfa84fa0643" containerName="manager"
Oct 30 03:04:50.607758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.607741    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-vx8p5"
Oct 30 03:04:50.630805 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.630773    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Oct 30 03:04:50.630979 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.630870    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-44l7r\""
Oct 30 03:04:50.632059 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.632034    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-vx8p5"]
Oct 30 03:04:50.729479 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.729440    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32315874-0935-4f77-8d57-6fe1983d0f0d-tls-certs\") pod \"model-serving-api-56ddc4c448-vx8p5\" (UID: \"32315874-0935-4f77-8d57-6fe1983d0f0d\") " pod="kserve/model-serving-api-56ddc4c448-vx8p5"
Oct 30 03:04:50.729711 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.729507    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl8vj\" (UniqueName: \"kubernetes.io/projected/32315874-0935-4f77-8d57-6fe1983d0f0d-kube-api-access-hl8vj\") pod \"model-serving-api-56ddc4c448-vx8p5\" (UID: \"32315874-0935-4f77-8d57-6fe1983d0f0d\") " pod="kserve/model-serving-api-56ddc4c448-vx8p5"
Oct 30 03:04:50.830836 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.830799    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32315874-0935-4f77-8d57-6fe1983d0f0d-tls-certs\") pod \"model-serving-api-56ddc4c448-vx8p5\" (UID: \"32315874-0935-4f77-8d57-6fe1983d0f0d\") " pod="kserve/model-serving-api-56ddc4c448-vx8p5"
Oct 30 03:04:50.831006 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.830853    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl8vj\" (UniqueName: \"kubernetes.io/projected/32315874-0935-4f77-8d57-6fe1983d0f0d-kube-api-access-hl8vj\") pod \"model-serving-api-56ddc4c448-vx8p5\" (UID: \"32315874-0935-4f77-8d57-6fe1983d0f0d\") " pod="kserve/model-serving-api-56ddc4c448-vx8p5"
Oct 30 03:04:50.833559 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.833528    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32315874-0935-4f77-8d57-6fe1983d0f0d-tls-certs\") pod \"model-serving-api-56ddc4c448-vx8p5\" (UID: \"32315874-0935-4f77-8d57-6fe1983d0f0d\") " pod="kserve/model-serving-api-56ddc4c448-vx8p5"
Oct 30 03:04:50.852310 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.852280    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl8vj\" (UniqueName: \"kubernetes.io/projected/32315874-0935-4f77-8d57-6fe1983d0f0d-kube-api-access-hl8vj\") pod \"model-serving-api-56ddc4c448-vx8p5\" (UID: \"32315874-0935-4f77-8d57-6fe1983d0f0d\") " pod="kserve/model-serving-api-56ddc4c448-vx8p5"
Oct 30 03:04:50.918480 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:50.918392    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-vx8p5"
Oct 30 03:04:51.210215 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:51.210109    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-vx8p5"]
Oct 30 03:04:51.212949 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:04:51.212920    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32315874_0935_4f77_8d57_6fe1983d0f0d.slice/crio-21fdd583ecd28c864c9fe3c69ce4fe28dc3d1d51fe4ec3ed11d2b339ec08a636 WatchSource:0}: Error finding container 21fdd583ecd28c864c9fe3c69ce4fe28dc3d1d51fe4ec3ed11d2b339ec08a636: Status 404 returned error can't find the container with id 21fdd583ecd28c864c9fe3c69ce4fe28dc3d1d51fe4ec3ed11d2b339ec08a636
Oct 30 03:04:51.882810 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:51.882772    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-vx8p5" event={"ID":"32315874-0935-4f77-8d57-6fe1983d0f0d","Type":"ContainerStarted","Data":"21fdd583ecd28c864c9fe3c69ce4fe28dc3d1d51fe4ec3ed11d2b339ec08a636"}
Oct 30 03:04:52.891221 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:52.891136    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-vx8p5" event={"ID":"32315874-0935-4f77-8d57-6fe1983d0f0d","Type":"ContainerStarted","Data":"a4a17abdecc29d0e87ce6f52f6ab039bbb5f37de994d7bdfa7743c00f477023a"}
Oct 30 03:04:52.891221 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:52.891199    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-56ddc4c448-vx8p5"
Oct 30 03:04:52.928622 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:52.928576    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/model-serving-api-56ddc4c448-vx8p5" podStartSLOduration=1.502173859 podStartE2EDuration="2.928561598s" totalImagesPullingTime="1.426387739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:52.928015251 +0000 UTC m=+598.438184279" watchObservedRunningTime="2026-10-30 03:04:52.928561598 +0000 UTC m=+598.438730618"
Oct 30 03:04:54.663461 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:54.663421    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:04:54.664110 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:54.664093    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:04:54.668399 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:54.668379    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:04:54.668891 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:04:54.668875    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:04:55.921432 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:04:55.921399    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:05:03.899195 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:03.899164    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-56ddc4c448-vx8p5"
Oct 30 03:05:06.106245 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.106215    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-init-4phnn"]
Oct 30 03:05:06.108992 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.108970    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4phnn"
Oct 30 03:05:06.122097 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.122076    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 30 03:05:06.133805 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.133782    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4phnn"]
Oct 30 03:05:06.145062 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.145040    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfae1ca1-3431-419f-9723-b428a8322ff7-model-cache\") pod \"s3-init-4phnn\" (UID: \"cfae1ca1-3431-419f-9723-b428a8322ff7\") " pod="kserve/s3-init-4phnn"
Oct 30 03:05:06.145139 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.145065    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkgg2\" (UniqueName: \"kubernetes.io/projected/cfae1ca1-3431-419f-9723-b428a8322ff7-kube-api-access-hkgg2\") pod \"s3-init-4phnn\" (UID: \"cfae1ca1-3431-419f-9723-b428a8322ff7\") " pod="kserve/s3-init-4phnn"
Oct 30 03:05:06.245386 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.245352    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfae1ca1-3431-419f-9723-b428a8322ff7-model-cache\") pod \"s3-init-4phnn\" (UID: \"cfae1ca1-3431-419f-9723-b428a8322ff7\") " pod="kserve/s3-init-4phnn"
Oct 30 03:05:06.245386 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.245390    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgg2\" (UniqueName: \"kubernetes.io/projected/cfae1ca1-3431-419f-9723-b428a8322ff7-kube-api-access-hkgg2\") pod \"s3-init-4phnn\" (UID: \"cfae1ca1-3431-419f-9723-b428a8322ff7\") " pod="kserve/s3-init-4phnn"
Oct 30 03:05:06.245731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.245710    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfae1ca1-3431-419f-9723-b428a8322ff7-model-cache\") pod \"s3-init-4phnn\" (UID: \"cfae1ca1-3431-419f-9723-b428a8322ff7\") " pod="kserve/s3-init-4phnn"
Oct 30 03:05:06.267568 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.267537    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkgg2\" (UniqueName: \"kubernetes.io/projected/cfae1ca1-3431-419f-9723-b428a8322ff7-kube-api-access-hkgg2\") pod \"s3-init-4phnn\" (UID: \"cfae1ca1-3431-419f-9723-b428a8322ff7\") " pod="kserve/s3-init-4phnn"
Oct 30 03:05:06.429425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.429346    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4phnn"
Oct 30 03:05:06.726906 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.726875    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4phnn"]
Oct 30 03:05:06.729841 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:05:06.729811    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfae1ca1_3431_419f_9723_b428a8322ff7.slice/crio-9da28439056e7ec586e8e1062cc309bc8e85f1c40415c97a73dee4fdb21bf0db WatchSource:0}: Error finding container 9da28439056e7ec586e8e1062cc309bc8e85f1c40415c97a73dee4fdb21bf0db: Status 404 returned error can't find the container with id 9da28439056e7ec586e8e1062cc309bc8e85f1c40415c97a73dee4fdb21bf0db
Oct 30 03:05:06.951498 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:06.951466    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4phnn" event={"ID":"cfae1ca1-3431-419f-9723-b428a8322ff7","Type":"ContainerStarted","Data":"9da28439056e7ec586e8e1062cc309bc8e85f1c40415c97a73dee4fdb21bf0db"}
Oct 30 03:05:10.969482 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:10.969444    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4phnn" event={"ID":"cfae1ca1-3431-419f-9723-b428a8322ff7","Type":"ContainerStarted","Data":"2f9ba9c1fa476c1fb7444ec9e7ae9d7e3279f7e2ab7cc0f9e2a1656755db0e78"}
Oct 30 03:05:11.975337 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:11.975306    4885 generic.go:356] "Generic (PLEG): container finished" podID="cfae1ca1-3431-419f-9723-b428a8322ff7" containerID="2f9ba9c1fa476c1fb7444ec9e7ae9d7e3279f7e2ab7cc0f9e2a1656755db0e78" exitCode=0
Oct 30 03:05:11.975770 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:11.975373    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4phnn" event={"ID":"cfae1ca1-3431-419f-9723-b428a8322ff7","Type":"ContainerDied","Data":"2f9ba9c1fa476c1fb7444ec9e7ae9d7e3279f7e2ab7cc0f9e2a1656755db0e78"}
Oct 30 03:05:17.001407 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:17.001372    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4phnn" event={"ID":"cfae1ca1-3431-419f-9723-b428a8322ff7","Type":"ContainerStarted","Data":"d9700bdc972dcc27f312fed6091d26cbbb7e91a342913883cbc73c7b580a5377"}
Oct 30 03:05:17.039300 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:05:17.039251    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-init-4phnn" podStartSLOduration=1.697684776 podStartE2EDuration="11.039237107s" totalImagesPullingTime="8.151303417s" totalInitContainerRuntime="1.190248914s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:17.038846649 +0000 UTC m=+622.549015687" watchObservedRunningTime="2026-10-30 03:05:17.039237107 +0000 UTC m=+622.549406133"
Oct 30 03:05:25.921540 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:05:25.921498    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:05:55.921110 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:05:55.921072    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:06:22.260182 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:22.260149    4885 generic.go:356] "Generic (PLEG): container finished" podID="cfae1ca1-3431-419f-9723-b428a8322ff7" containerID="d9700bdc972dcc27f312fed6091d26cbbb7e91a342913883cbc73c7b580a5377" exitCode=0
Oct 30 03:06:22.260594 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:22.260224    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4phnn" event={"ID":"cfae1ca1-3431-419f-9723-b428a8322ff7","Type":"ContainerDied","Data":"d9700bdc972dcc27f312fed6091d26cbbb7e91a342913883cbc73c7b580a5377"}
Oct 30 03:06:23.389432 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:23.389412    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4phnn"
Oct 30 03:06:23.434998 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:23.434976    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfae1ca1-3431-419f-9723-b428a8322ff7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfae1ca1-3431-419f-9723-b428a8322ff7-model-cache\") pod \"cfae1ca1-3431-419f-9723-b428a8322ff7\" (UID: \"cfae1ca1-3431-419f-9723-b428a8322ff7\") "
Oct 30 03:06:23.435099 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:23.435019    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfae1ca1-3431-419f-9723-b428a8322ff7-kube-api-access-hkgg2\" (UniqueName: \"kubernetes.io/projected/cfae1ca1-3431-419f-9723-b428a8322ff7-kube-api-access-hkgg2\") pod \"cfae1ca1-3431-419f-9723-b428a8322ff7\" (UID: \"cfae1ca1-3431-419f-9723-b428a8322ff7\") "
Oct 30 03:06:23.437122 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:23.437097    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfae1ca1-3431-419f-9723-b428a8322ff7-kube-api-access-hkgg2" pod "cfae1ca1-3431-419f-9723-b428a8322ff7" (UID: "cfae1ca1-3431-419f-9723-b428a8322ff7"). InnerVolumeSpecName "kube-api-access-hkgg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:23.460137 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:23.460095    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfae1ca1-3431-419f-9723-b428a8322ff7-model-cache" pod "cfae1ca1-3431-419f-9723-b428a8322ff7" (UID: "cfae1ca1-3431-419f-9723-b428a8322ff7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:23.536224 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:23.536171    4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfae1ca1-3431-419f-9723-b428a8322ff7-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:06:23.536224 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:23.536193    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkgg2\" (UniqueName: \"kubernetes.io/projected/cfae1ca1-3431-419f-9723-b428a8322ff7-kube-api-access-hkgg2\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:06:24.269462 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:24.269430    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4phnn"
Oct 30 03:06:24.269652 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:24.269430    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4phnn" event={"ID":"cfae1ca1-3431-419f-9723-b428a8322ff7","Type":"ContainerDied","Data":"9da28439056e7ec586e8e1062cc309bc8e85f1c40415c97a73dee4fdb21bf0db"}
Oct 30 03:06:24.269652 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:24.269553    4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9da28439056e7ec586e8e1062cc309bc8e85f1c40415c97a73dee4fdb21bf0db"
Oct 30 03:06:25.445465 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.445439    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"]
Oct 30 03:06:25.445871 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.445792    4885 cpu_manager.go:414] "containerMap: removing container" podUID="cfae1ca1-3431-419f-9723-b428a8322ff7" containerName="download-hf-model"
Oct 30 03:06:25.445871 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.445803    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfae1ca1-3431-419f-9723-b428a8322ff7" containerName="download-hf-model"
Oct 30 03:06:25.445871 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.445819    4885 cpu_manager.go:414] "containerMap: removing container" podUID="cfae1ca1-3431-419f-9723-b428a8322ff7" containerName="s3-init"
Oct 30 03:06:25.445871 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.445824    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfae1ca1-3431-419f-9723-b428a8322ff7" containerName="s3-init"
Oct 30 03:06:25.446034 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.445869    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfae1ca1-3431-419f-9723-b428a8322ff7" containerName="s3-init"
Oct 30 03:06:25.447791 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.447772    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"
Oct 30 03:06:25.460655 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.460634    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Oct 30 03:06:25.477402 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.477378    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"]
Oct 30 03:06:25.551564 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.551528    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31ff4beb-47f0-4583-8d9d-161437ad848d-data\") pod \"seaweedfs-tls-custom-64cc6669cb-nwg7j\" (UID: \"31ff4beb-47f0-4583-8d9d-161437ad848d\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"
Oct 30 03:06:25.551736 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.551577    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwxsb\" (UniqueName: \"kubernetes.io/projected/31ff4beb-47f0-4583-8d9d-161437ad848d-kube-api-access-pwxsb\") pod \"seaweedfs-tls-custom-64cc6669cb-nwg7j\" (UID: \"31ff4beb-47f0-4583-8d9d-161437ad848d\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"
Oct 30 03:06:25.652366 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.652334    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31ff4beb-47f0-4583-8d9d-161437ad848d-data\") pod \"seaweedfs-tls-custom-64cc6669cb-nwg7j\" (UID: \"31ff4beb-47f0-4583-8d9d-161437ad848d\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"
Oct 30 03:06:25.652524 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.652373    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwxsb\" (UniqueName: \"kubernetes.io/projected/31ff4beb-47f0-4583-8d9d-161437ad848d-kube-api-access-pwxsb\") pod \"seaweedfs-tls-custom-64cc6669cb-nwg7j\" (UID: \"31ff4beb-47f0-4583-8d9d-161437ad848d\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"
Oct 30 03:06:25.652694 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.652677    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31ff4beb-47f0-4583-8d9d-161437ad848d-data\") pod \"seaweedfs-tls-custom-64cc6669cb-nwg7j\" (UID: \"31ff4beb-47f0-4583-8d9d-161437ad848d\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"
Oct 30 03:06:25.674326 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.674301    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwxsb\" (UniqueName: \"kubernetes.io/projected/31ff4beb-47f0-4583-8d9d-161437ad848d-kube-api-access-pwxsb\") pod \"seaweedfs-tls-custom-64cc6669cb-nwg7j\" (UID: \"31ff4beb-47f0-4583-8d9d-161437ad848d\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"
Oct 30 03:06:25.757858 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:25.757836    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"
Oct 30 03:06:25.922083 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:06:25.922048    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:06:26.045947 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:26.045908    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"]
Oct 30 03:06:26.048713 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:06:26.048687    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ff4beb_47f0_4583_8d9d_161437ad848d.slice/crio-d278eb43305e5109aa61b8177373d09e83d717e348d14a610222340092f962d7 WatchSource:0}: Error finding container d278eb43305e5109aa61b8177373d09e83d717e348d14a610222340092f962d7: Status 404 returned error can't find the container with id d278eb43305e5109aa61b8177373d09e83d717e348d14a610222340092f962d7
Oct 30 03:06:26.278098 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:26.278058    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j" event={"ID":"31ff4beb-47f0-4583-8d9d-161437ad848d","Type":"ContainerStarted","Data":"d278eb43305e5109aa61b8177373d09e83d717e348d14a610222340092f962d7"}
Oct 30 03:06:27.283286 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:27.283255    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j" event={"ID":"31ff4beb-47f0-4583-8d9d-161437ad848d","Type":"ContainerStarted","Data":"6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319"}
Oct 30 03:06:27.320422 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:27.320376    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j" podStartSLOduration=2.074813038 podStartE2EDuration="2.320363386s" totalImagesPullingTime="245.550348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:27.319133138 +0000 UTC m=+692.829302167" watchObservedRunningTime="2026-10-30 03:06:27.320363386 +0000 UTC m=+692.830532446"
Oct 30 03:06:28.513152 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:28.513120    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"]
Oct 30 03:06:29.291985 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:29.291928    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j" podUID="31ff4beb-47f0-4583-8d9d-161437ad848d" containerName="seaweedfs-tls-custom" containerID="cri-o://6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319" gracePeriod=30
Oct 30 03:06:30.529964 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.529942    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"
Oct 30 03:06:30.588627 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.588528    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ff4beb-47f0-4583-8d9d-161437ad848d-kube-api-access-pwxsb\" (UniqueName: \"kubernetes.io/projected/31ff4beb-47f0-4583-8d9d-161437ad848d-kube-api-access-pwxsb\") pod \"31ff4beb-47f0-4583-8d9d-161437ad848d\" (UID: \"31ff4beb-47f0-4583-8d9d-161437ad848d\") "
Oct 30 03:06:30.588776 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.588668    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ff4beb-47f0-4583-8d9d-161437ad848d-data\" (UniqueName: \"kubernetes.io/empty-dir/31ff4beb-47f0-4583-8d9d-161437ad848d-data\") pod \"31ff4beb-47f0-4583-8d9d-161437ad848d\" (UID: \"31ff4beb-47f0-4583-8d9d-161437ad848d\") "
Oct 30 03:06:30.590037 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.590000    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ff4beb-47f0-4583-8d9d-161437ad848d-data" pod "31ff4beb-47f0-4583-8d9d-161437ad848d" (UID: "31ff4beb-47f0-4583-8d9d-161437ad848d"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:30.590850 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.590819    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ff4beb-47f0-4583-8d9d-161437ad848d-kube-api-access-pwxsb" pod "31ff4beb-47f0-4583-8d9d-161437ad848d" (UID: "31ff4beb-47f0-4583-8d9d-161437ad848d"). InnerVolumeSpecName "kube-api-access-pwxsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:30.603657 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.603608    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"]
Oct 30 03:06:30.604034 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.604020    4885 cpu_manager.go:414] "containerMap: removing container" podUID="31ff4beb-47f0-4583-8d9d-161437ad848d" containerName="seaweedfs-tls-custom"
Oct 30 03:06:30.604078 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.604035    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ff4beb-47f0-4583-8d9d-161437ad848d" containerName="seaweedfs-tls-custom"
Oct 30 03:06:30.604132 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.604121    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ff4beb-47f0-4583-8d9d-161437ad848d" containerName="seaweedfs-tls-custom"
Oct 30 03:06:30.606752 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.606736    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"
Oct 30 03:06:30.620607 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.620583    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Oct 30 03:06:30.625799 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.625774    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"]
Oct 30 03:06:30.689909 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.689871    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/4f051988-5b08-49b8-ac92-7a7869fd9f19-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-2qzpc\" (UID: \"4f051988-5b08-49b8-ac92-7a7869fd9f19\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"
Oct 30 03:06:30.690091 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.689926    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f051988-5b08-49b8-ac92-7a7869fd9f19-data\") pod \"seaweedfs-tls-custom-685c9f986c-2qzpc\" (UID: \"4f051988-5b08-49b8-ac92-7a7869fd9f19\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"
Oct 30 03:06:30.690177 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.690153    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6s42\" (UniqueName: \"kubernetes.io/projected/4f051988-5b08-49b8-ac92-7a7869fd9f19-kube-api-access-t6s42\") pod \"seaweedfs-tls-custom-685c9f986c-2qzpc\" (UID: \"4f051988-5b08-49b8-ac92-7a7869fd9f19\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"
Oct 30 03:06:30.690229 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.690218    4885 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31ff4beb-47f0-4583-8d9d-161437ad848d-data\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:06:30.690264 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.690232    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwxsb\" (UniqueName: \"kubernetes.io/projected/31ff4beb-47f0-4583-8d9d-161437ad848d-kube-api-access-pwxsb\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:06:30.791281 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.791237    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f051988-5b08-49b8-ac92-7a7869fd9f19-data\") pod \"seaweedfs-tls-custom-685c9f986c-2qzpc\" (UID: \"4f051988-5b08-49b8-ac92-7a7869fd9f19\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"
Oct 30 03:06:30.791482 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.791293    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6s42\" (UniqueName: \"kubernetes.io/projected/4f051988-5b08-49b8-ac92-7a7869fd9f19-kube-api-access-t6s42\") pod \"seaweedfs-tls-custom-685c9f986c-2qzpc\" (UID: \"4f051988-5b08-49b8-ac92-7a7869fd9f19\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"
Oct 30 03:06:30.791482 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.791344    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/4f051988-5b08-49b8-ac92-7a7869fd9f19-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-2qzpc\" (UID: \"4f051988-5b08-49b8-ac92-7a7869fd9f19\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"
Oct 30 03:06:30.791712 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.791691    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f051988-5b08-49b8-ac92-7a7869fd9f19-data\") pod \"seaweedfs-tls-custom-685c9f986c-2qzpc\" (UID: \"4f051988-5b08-49b8-ac92-7a7869fd9f19\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"
Oct 30 03:06:30.793984 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.793960    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/4f051988-5b08-49b8-ac92-7a7869fd9f19-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-2qzpc\" (UID: \"4f051988-5b08-49b8-ac92-7a7869fd9f19\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"
Oct 30 03:06:30.813135 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.813109    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6s42\" (UniqueName: \"kubernetes.io/projected/4f051988-5b08-49b8-ac92-7a7869fd9f19-kube-api-access-t6s42\") pod \"seaweedfs-tls-custom-685c9f986c-2qzpc\" (UID: \"4f051988-5b08-49b8-ac92-7a7869fd9f19\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"
Oct 30 03:06:30.925735 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:30.925660    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"
Oct 30 03:06:31.215644 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:31.215420    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-2qzpc"]
Oct 30 03:06:31.218203 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:06:31.218180    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f051988_5b08_49b8_ac92_7a7869fd9f19.slice/crio-bebce3921850689718483a41163fd7cc74131c4395ce71b06370ce590c33cb32 WatchSource:0}: Error finding container bebce3921850689718483a41163fd7cc74131c4395ce71b06370ce590c33cb32: Status 404 returned error can't find the container with id bebce3921850689718483a41163fd7cc74131c4395ce71b06370ce590c33cb32
Oct 30 03:06:31.301172 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:31.301145    4885 generic.go:356] "Generic (PLEG): container finished" podID="31ff4beb-47f0-4583-8d9d-161437ad848d" containerID="6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319" exitCode=0
Oct 30 03:06:31.301367 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:31.301191    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"
Oct 30 03:06:31.301367 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:31.301241    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j" event={"ID":"31ff4beb-47f0-4583-8d9d-161437ad848d","Type":"ContainerDied","Data":"6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319"}
Oct 30 03:06:31.301367 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:31.301286    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j" event={"ID":"31ff4beb-47f0-4583-8d9d-161437ad848d","Type":"ContainerDied","Data":"d278eb43305e5109aa61b8177373d09e83d717e348d14a610222340092f962d7"}
Oct 30 03:06:31.301367 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:31.301307    4885 scope.go:122] "RemoveContainer" containerID="6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319"
Oct 30 03:06:31.302530 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:31.302502    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc" event={"ID":"4f051988-5b08-49b8-ac92-7a7869fd9f19","Type":"ContainerStarted","Data":"bebce3921850689718483a41163fd7cc74131c4395ce71b06370ce590c33cb32"}
Oct 30 03:06:31.317125 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:31.317100    4885 scope.go:122] "RemoveContainer" containerID="6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319"
Oct 30 03:06:31.317402 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:06:31.317386    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319\": container with ID starting with 6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319 not found: ID does not exist" containerID="6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319"
Oct 30 03:06:31.317446 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:31.317410    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319"} err="failed to get container status \"6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319\": rpc error: code = NotFound desc = could not find container \"6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319\": container with ID starting with 6a8ebbbdee34c3be4a8d1df047e7e32d0c059c0be85a6cac8d6d68883ebd3319 not found: ID does not exist"
Oct 30 03:06:31.346129 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:31.346103    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"]
Oct 30 03:06:31.355792 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:31.355766    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-nwg7j"]
Oct 30 03:06:32.308464 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:32.308434    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc" event={"ID":"4f051988-5b08-49b8-ac92-7a7869fd9f19","Type":"ContainerStarted","Data":"01dce948bdb638634b8f32e0f6fe22bd1f1bde1f139dff54b9db5adc4739f03c"}
Oct 30 03:06:32.345324 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:32.345271    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-685c9f986c-2qzpc" podStartSLOduration=2.05267447 podStartE2EDuration="2.345258957s" totalImagesPullingTime="292.584487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:32.344594188 +0000 UTC m=+697.854763232" watchObservedRunningTime="2026-10-30 03:06:32.345258957 +0000 UTC m=+697.855428015"
Oct 30 03:06:32.624118 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:32.624033    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ff4beb-47f0-4583-8d9d-161437ad848d" path="/var/lib/kubelet/pods/31ff4beb-47f0-4583-8d9d-161437ad848d/volumes"
Oct 30 03:06:34.877233 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:34.877208    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-5rlhg"]
Oct 30 03:06:34.880955 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:34.880931    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-5rlhg"
Oct 30 03:06:34.893816 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:34.893799    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 30 03:06:34.904953 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:34.904933    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-5rlhg"]
Oct 30 03:06:34.921609 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:34.921586    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e572c5b9-7c49-4682-8305-e9f61a5ead70-model-cache\") pod \"s3-tls-init-custom-5rlhg\" (UID: \"e572c5b9-7c49-4682-8305-e9f61a5ead70\") " pod="kserve/s3-tls-init-custom-5rlhg"
Oct 30 03:06:34.921724 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:34.921704    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kr6q\" (UniqueName: \"kubernetes.io/projected/e572c5b9-7c49-4682-8305-e9f61a5ead70-kube-api-access-9kr6q\") pod \"s3-tls-init-custom-5rlhg\" (UID: \"e572c5b9-7c49-4682-8305-e9f61a5ead70\") " pod="kserve/s3-tls-init-custom-5rlhg"
Oct 30 03:06:35.022950 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:35.022906    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kr6q\" (UniqueName: \"kubernetes.io/projected/e572c5b9-7c49-4682-8305-e9f61a5ead70-kube-api-access-9kr6q\") pod \"s3-tls-init-custom-5rlhg\" (UID: \"e572c5b9-7c49-4682-8305-e9f61a5ead70\") " pod="kserve/s3-tls-init-custom-5rlhg"
Oct 30 03:06:35.023089 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:35.022970    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e572c5b9-7c49-4682-8305-e9f61a5ead70-model-cache\") pod \"s3-tls-init-custom-5rlhg\" (UID: \"e572c5b9-7c49-4682-8305-e9f61a5ead70\") " pod="kserve/s3-tls-init-custom-5rlhg"
Oct 30 03:06:35.023297 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:35.023282    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e572c5b9-7c49-4682-8305-e9f61a5ead70-model-cache\") pod \"s3-tls-init-custom-5rlhg\" (UID: \"e572c5b9-7c49-4682-8305-e9f61a5ead70\") " pod="kserve/s3-tls-init-custom-5rlhg"
Oct 30 03:06:35.044743 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:35.044716    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kr6q\" (UniqueName: \"kubernetes.io/projected/e572c5b9-7c49-4682-8305-e9f61a5ead70-kube-api-access-9kr6q\") pod \"s3-tls-init-custom-5rlhg\" (UID: \"e572c5b9-7c49-4682-8305-e9f61a5ead70\") " pod="kserve/s3-tls-init-custom-5rlhg"
Oct 30 03:06:35.203641 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:35.203550    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-5rlhg"
Oct 30 03:06:35.492672 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:35.492641    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-5rlhg"]
Oct 30 03:06:35.495794 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:06:35.495766    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode572c5b9_7c49_4682_8305_e9f61a5ead70.slice/crio-56a96b2874ab25669c9bd256ea09a6ef3105349357dded5ae79af7a4b55ef404 WatchSource:0}: Error finding container 56a96b2874ab25669c9bd256ea09a6ef3105349357dded5ae79af7a4b55ef404: Status 404 returned error can't find the container with id 56a96b2874ab25669c9bd256ea09a6ef3105349357dded5ae79af7a4b55ef404
Oct 30 03:06:36.329436 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:36.329399    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5rlhg" event={"ID":"e572c5b9-7c49-4682-8305-e9f61a5ead70","Type":"ContainerStarted","Data":"02af9ff574f2e7a86882f08fd34ed67db54726a9b25a09728b108ad906dc9de6"}
Oct 30 03:06:36.329848 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:36.329443    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5rlhg" event={"ID":"e572c5b9-7c49-4682-8305-e9f61a5ead70","Type":"ContainerStarted","Data":"56a96b2874ab25669c9bd256ea09a6ef3105349357dded5ae79af7a4b55ef404"}
Oct 30 03:06:37.334860 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:37.334773    4885 generic.go:356] "Generic (PLEG): container finished" podID="e572c5b9-7c49-4682-8305-e9f61a5ead70" containerID="02af9ff574f2e7a86882f08fd34ed67db54726a9b25a09728b108ad906dc9de6" exitCode=0
Oct 30 03:06:37.335214 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:37.334863    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5rlhg" event={"ID":"e572c5b9-7c49-4682-8305-e9f61a5ead70","Type":"ContainerDied","Data":"02af9ff574f2e7a86882f08fd34ed67db54726a9b25a09728b108ad906dc9de6"}
Oct 30 03:06:38.340852 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:38.340814    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5rlhg" event={"ID":"e572c5b9-7c49-4682-8305-e9f61a5ead70","Type":"ContainerStarted","Data":"5e5763dc7b6e207e3ecc92ebc235e6fb65dd1dd5d4a35ee658da6bb47b1df2e8"}
Oct 30 03:06:38.379082 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:38.379029    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-5rlhg" podStartSLOduration=2.9166205 podStartE2EDuration="4.37901521s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.46239471s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:38.378149171 +0000 UTC m=+703.888318198" watchObservedRunningTime="2026-10-30 03:06:38.37901521 +0000 UTC m=+703.889184237"
Oct 30 03:06:43.371974 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:43.371888    4885 generic.go:356] "Generic (PLEG): container finished" podID="e572c5b9-7c49-4682-8305-e9f61a5ead70" containerID="5e5763dc7b6e207e3ecc92ebc235e6fb65dd1dd5d4a35ee658da6bb47b1df2e8" exitCode=0
Oct 30 03:06:43.372379 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:43.371966    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5rlhg" event={"ID":"e572c5b9-7c49-4682-8305-e9f61a5ead70","Type":"ContainerDied","Data":"5e5763dc7b6e207e3ecc92ebc235e6fb65dd1dd5d4a35ee658da6bb47b1df2e8"}
Oct 30 03:06:44.510685 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:44.510657    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-5rlhg"
Oct 30 03:06:44.600824 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:44.600785    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e572c5b9-7c49-4682-8305-e9f61a5ead70-kube-api-access-9kr6q\" (UniqueName: \"kubernetes.io/projected/e572c5b9-7c49-4682-8305-e9f61a5ead70-kube-api-access-9kr6q\") pod \"e572c5b9-7c49-4682-8305-e9f61a5ead70\" (UID: \"e572c5b9-7c49-4682-8305-e9f61a5ead70\") "
Oct 30 03:06:44.601071 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:44.600872    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e572c5b9-7c49-4682-8305-e9f61a5ead70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e572c5b9-7c49-4682-8305-e9f61a5ead70-model-cache\") pod \"e572c5b9-7c49-4682-8305-e9f61a5ead70\" (UID: \"e572c5b9-7c49-4682-8305-e9f61a5ead70\") "
Oct 30 03:06:44.602980 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:44.602954    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e572c5b9-7c49-4682-8305-e9f61a5ead70-kube-api-access-9kr6q" pod "e572c5b9-7c49-4682-8305-e9f61a5ead70" (UID: "e572c5b9-7c49-4682-8305-e9f61a5ead70"). InnerVolumeSpecName "kube-api-access-9kr6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:44.621770 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:44.621737    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e572c5b9-7c49-4682-8305-e9f61a5ead70-model-cache" pod "e572c5b9-7c49-4682-8305-e9f61a5ead70" (UID: "e572c5b9-7c49-4682-8305-e9f61a5ead70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:44.701475 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:44.701441    4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e572c5b9-7c49-4682-8305-e9f61a5ead70-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:06:44.701475 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:44.701469    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kr6q\" (UniqueName: \"kubernetes.io/projected/e572c5b9-7c49-4682-8305-e9f61a5ead70-kube-api-access-9kr6q\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:06:45.381942 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:45.381902    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5rlhg" event={"ID":"e572c5b9-7c49-4682-8305-e9f61a5ead70","Type":"ContainerDied","Data":"56a96b2874ab25669c9bd256ea09a6ef3105349357dded5ae79af7a4b55ef404"}
Oct 30 03:06:45.381942 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:45.381943    4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56a96b2874ab25669c9bd256ea09a6ef3105349357dded5ae79af7a4b55ef404"
Oct 30 03:06:45.382217 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:45.381943    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-5rlhg"
Oct 30 03:06:51.044275 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.044246    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-c2chh"]
Oct 30 03:06:51.044818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.044718    4885 cpu_manager.go:414] "containerMap: removing container" podUID="e572c5b9-7c49-4682-8305-e9f61a5ead70" containerName="download-hf-model"
Oct 30 03:06:51.044818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.044737    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e572c5b9-7c49-4682-8305-e9f61a5ead70" containerName="download-hf-model"
Oct 30 03:06:51.044818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.044751    4885 cpu_manager.go:414] "containerMap: removing container" podUID="e572c5b9-7c49-4682-8305-e9f61a5ead70" containerName="s3-tls-init-custom"
Oct 30 03:06:51.044818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.044759    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e572c5b9-7c49-4682-8305-e9f61a5ead70" containerName="s3-tls-init-custom"
Oct 30 03:06:51.045034 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.044864    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="e572c5b9-7c49-4682-8305-e9f61a5ead70" containerName="s3-tls-init-custom"
Oct 30 03:06:51.047181 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.047161    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-c2chh"
Oct 30 03:06:51.059748 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.059732    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 30 03:06:51.059877 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.059854    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Oct 30 03:06:51.071138 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.071114    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-c2chh"]
Oct 30 03:06:51.151034 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.151007    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqwbc\" (UniqueName: \"kubernetes.io/projected/bc1a4aab-0469-4563-ac49-77f4c6e419fa-kube-api-access-wqwbc\") pod \"s3-tls-init-serving-c2chh\" (UID: \"bc1a4aab-0469-4563-ac49-77f4c6e419fa\") " pod="kserve/s3-tls-init-serving-c2chh"
Oct 30 03:06:51.151179 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.151054    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1a4aab-0469-4563-ac49-77f4c6e419fa-model-cache\") pod \"s3-tls-init-serving-c2chh\" (UID: \"bc1a4aab-0469-4563-ac49-77f4c6e419fa\") " pod="kserve/s3-tls-init-serving-c2chh"
Oct 30 03:06:51.251598 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.251565    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1a4aab-0469-4563-ac49-77f4c6e419fa-model-cache\") pod \"s3-tls-init-serving-c2chh\" (UID: \"bc1a4aab-0469-4563-ac49-77f4c6e419fa\") " pod="kserve/s3-tls-init-serving-c2chh"
Oct 30 03:06:51.251749 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.251653    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqwbc\" (UniqueName: \"kubernetes.io/projected/bc1a4aab-0469-4563-ac49-77f4c6e419fa-kube-api-access-wqwbc\") pod \"s3-tls-init-serving-c2chh\" (UID: \"bc1a4aab-0469-4563-ac49-77f4c6e419fa\") " pod="kserve/s3-tls-init-serving-c2chh"
Oct 30 03:06:51.251979 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.251960    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1a4aab-0469-4563-ac49-77f4c6e419fa-model-cache\") pod \"s3-tls-init-serving-c2chh\" (UID: \"bc1a4aab-0469-4563-ac49-77f4c6e419fa\") " pod="kserve/s3-tls-init-serving-c2chh"
Oct 30 03:06:51.273438 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.273408    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqwbc\" (UniqueName: \"kubernetes.io/projected/bc1a4aab-0469-4563-ac49-77f4c6e419fa-kube-api-access-wqwbc\") pod \"s3-tls-init-serving-c2chh\" (UID: \"bc1a4aab-0469-4563-ac49-77f4c6e419fa\") " pod="kserve/s3-tls-init-serving-c2chh"
Oct 30 03:06:51.373146 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.373077    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-c2chh"
Oct 30 03:06:51.665251 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:51.665081    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-c2chh"]
Oct 30 03:06:51.668044 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:06:51.668019    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc1a4aab_0469_4563_ac49_77f4c6e419fa.slice/crio-7d7d69e2ff2fdaa372a1d813e68ce585c732ba73232e2269973e10ac315d4d9b WatchSource:0}: Error finding container 7d7d69e2ff2fdaa372a1d813e68ce585c732ba73232e2269973e10ac315d4d9b: Status 404 returned error can't find the container with id 7d7d69e2ff2fdaa372a1d813e68ce585c732ba73232e2269973e10ac315d4d9b
Oct 30 03:06:52.413086 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:52.413053    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-c2chh" event={"ID":"bc1a4aab-0469-4563-ac49-77f4c6e419fa","Type":"ContainerStarted","Data":"2bb4dd550aeedc39c6784e195974026211d020418bf757ebfad7b4fdb1ae5cba"}
Oct 30 03:06:52.413086 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:52.413099    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-c2chh" event={"ID":"bc1a4aab-0469-4563-ac49-77f4c6e419fa","Type":"ContainerStarted","Data":"7d7d69e2ff2fdaa372a1d813e68ce585c732ba73232e2269973e10ac315d4d9b"}
Oct 30 03:06:53.418912 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:53.418879    4885 generic.go:356] "Generic (PLEG): container finished" podID="bc1a4aab-0469-4563-ac49-77f4c6e419fa" containerID="2bb4dd550aeedc39c6784e195974026211d020418bf757ebfad7b4fdb1ae5cba" exitCode=0
Oct 30 03:06:53.418912 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:53.418915    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-c2chh" event={"ID":"bc1a4aab-0469-4563-ac49-77f4c6e419fa","Type":"ContainerDied","Data":"2bb4dd550aeedc39c6784e195974026211d020418bf757ebfad7b4fdb1ae5cba"}
Oct 30 03:06:54.424639 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:54.424580    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-c2chh" event={"ID":"bc1a4aab-0469-4563-ac49-77f4c6e419fa","Type":"ContainerStarted","Data":"2d0f519516fa274e76ad67e26364f3f9db004c52cdbe0d6074fa175c938d4bf7"}
Oct 30 03:06:54.463148 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:54.463098    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-c2chh" podStartSLOduration=2.25790971 podStartE2EDuration="3.463083271s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.205173561s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:54.46223656 +0000 UTC m=+719.972405606" watchObservedRunningTime="2026-10-30 03:06:54.463083271 +0000 UTC m=+719.973252328"
Oct 30 03:06:55.922387 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:06:55.922350    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:06:59.448805 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:59.448772    4885 generic.go:356] "Generic (PLEG): container finished" podID="bc1a4aab-0469-4563-ac49-77f4c6e419fa" containerID="2d0f519516fa274e76ad67e26364f3f9db004c52cdbe0d6074fa175c938d4bf7" exitCode=0
Oct 30 03:06:59.449115 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:06:59.448821    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-c2chh" event={"ID":"bc1a4aab-0469-4563-ac49-77f4c6e419fa","Type":"ContainerDied","Data":"2d0f519516fa274e76ad67e26364f3f9db004c52cdbe0d6074fa175c938d4bf7"}
Oct 30 03:07:00.589193 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:00.589165    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-c2chh"
Oct 30 03:07:00.732569 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:00.732544    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1a4aab-0469-4563-ac49-77f4c6e419fa-kube-api-access-wqwbc\" (UniqueName: \"kubernetes.io/projected/bc1a4aab-0469-4563-ac49-77f4c6e419fa-kube-api-access-wqwbc\") pod \"bc1a4aab-0469-4563-ac49-77f4c6e419fa\" (UID: \"bc1a4aab-0469-4563-ac49-77f4c6e419fa\") "
Oct 30 03:07:00.732740 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:00.732592    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1a4aab-0469-4563-ac49-77f4c6e419fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1a4aab-0469-4563-ac49-77f4c6e419fa-model-cache\") pod \"bc1a4aab-0469-4563-ac49-77f4c6e419fa\" (UID: \"bc1a4aab-0469-4563-ac49-77f4c6e419fa\") "
Oct 30 03:07:00.734758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:00.734731    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1a4aab-0469-4563-ac49-77f4c6e419fa-kube-api-access-wqwbc" pod "bc1a4aab-0469-4563-ac49-77f4c6e419fa" (UID: "bc1a4aab-0469-4563-ac49-77f4c6e419fa"). InnerVolumeSpecName "kube-api-access-wqwbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:00.754081 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:00.754048    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1a4aab-0469-4563-ac49-77f4c6e419fa-model-cache" pod "bc1a4aab-0469-4563-ac49-77f4c6e419fa" (UID: "bc1a4aab-0469-4563-ac49-77f4c6e419fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:00.833882 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:00.833856    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqwbc\" (UniqueName: \"kubernetes.io/projected/bc1a4aab-0469-4563-ac49-77f4c6e419fa-kube-api-access-wqwbc\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:07:00.833882 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:00.833878    4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1a4aab-0469-4563-ac49-77f4c6e419fa-model-cache\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:07:01.459707 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:01.459673    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-c2chh" event={"ID":"bc1a4aab-0469-4563-ac49-77f4c6e419fa","Type":"ContainerDied","Data":"7d7d69e2ff2fdaa372a1d813e68ce585c732ba73232e2269973e10ac315d4d9b"}
Oct 30 03:07:01.459707 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:01.459705    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-c2chh"
Oct 30 03:07:01.459707 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:01.459711    4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d7d69e2ff2fdaa372a1d813e68ce585c732ba73232e2269973e10ac315d4d9b"
Oct 30 03:07:22.186172 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.186137    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"]
Oct 30 03:07:22.186778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.186565    4885 cpu_manager.go:414] "containerMap: removing container" podUID="bc1a4aab-0469-4563-ac49-77f4c6e419fa" containerName="download-hf-model"
Oct 30 03:07:22.186778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.186584    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1a4aab-0469-4563-ac49-77f4c6e419fa" containerName="download-hf-model"
Oct 30 03:07:22.186778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.186652    4885 cpu_manager.go:414] "containerMap: removing container" podUID="bc1a4aab-0469-4563-ac49-77f4c6e419fa" containerName="s3-tls-init-serving"
Oct 30 03:07:22.186778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.186661    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1a4aab-0469-4563-ac49-77f4c6e419fa" containerName="s3-tls-init-serving"
Oct 30 03:07:22.186778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.186752    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1a4aab-0469-4563-ac49-77f4c6e419fa" containerName="s3-tls-init-serving"
Oct 30 03:07:22.190007 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.189991    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.203178 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.203150    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:07:22.213263 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.213238    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:07:22.213263 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.213256    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Oct 30 03:07:22.213425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.213255    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:07:22.213425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.213355    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Oct 30 03:07:22.213425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.213361    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:07:22.213425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.213415    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:07:22.216346 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.216325    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"]
Oct 30 03:07:22.300397 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.300367    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1aff33fb-25cf-4fe3-ab90-8b684710191e-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.300577 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.300408    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.300577 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.300519    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwsdl\" (UniqueName: \"kubernetes.io/projected/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-api-access-mwsdl\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.300577 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.300565    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.300737 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.300683    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1aff33fb-25cf-4fe3-ab90-8b684710191e-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.401769 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.401739    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwsdl\" (UniqueName: \"kubernetes.io/projected/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-api-access-mwsdl\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.401927 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.401788    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.401927 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.401824    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1aff33fb-25cf-4fe3-ab90-8b684710191e-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.401927 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.401853    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1aff33fb-25cf-4fe3-ab90-8b684710191e-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.401927 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.401877    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.402313 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.402293    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1aff33fb-25cf-4fe3-ab90-8b684710191e-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.402662 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.402645    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.402715 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.402651    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.404454 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.404430    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1aff33fb-25cf-4fe3-ab90-8b684710191e-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.423500 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.423481    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwsdl\" (UniqueName: \"kubernetes.io/projected/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-api-access-mwsdl\") pod \"isvc-sklearn-batcher-predictor-857545b8c4-qk6mg\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.482158 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.482135    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"]
Oct 30 03:07:22.486017 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.486002    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.499162 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.499112    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:07:22.499289 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.499249    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:07:22.499386 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.499361    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Oct 30 03:07:22.499516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.499382    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:07:22.499516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.499383    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Oct 30 03:07:22.499516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.499459    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:07:22.499701 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.499633    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:07:22.507809 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.507787    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"]
Oct 30 03:07:22.510435 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.510415    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:22.604228 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.604196    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b45c6f06-e176-4950-9230-8da4d7c60037-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.604386 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.604232    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b45c6f06-e176-4950-9230-8da4d7c60037-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.604386 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.604256    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.604492 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.604379    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf248\" (UniqueName: \"kubernetes.io/projected/b45c6f06-e176-4950-9230-8da4d7c60037-kube-api-access-pf248\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.604552 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.604484    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.705282 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.705250    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.705436 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.705301    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b45c6f06-e176-4950-9230-8da4d7c60037-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.705477 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.705431    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b45c6f06-e176-4950-9230-8da4d7c60037-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.705511 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.705472    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.705549 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.705527    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf248\" (UniqueName: \"kubernetes.io/projected/b45c6f06-e176-4950-9230-8da4d7c60037-kube-api-access-pf248\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.705843 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.705812    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b45c6f06-e176-4950-9230-8da4d7c60037-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.706081 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.706061    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.706166 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.706105    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.707944 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.707927    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b45c6f06-e176-4950-9230-8da4d7c60037-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.727185 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.727162    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf248\" (UniqueName: \"kubernetes.io/projected/b45c6f06-e176-4950-9230-8da4d7c60037-kube-api-access-pf248\") pod \"isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.803019 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.802952    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:22.818549 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:22.818513    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"]
Oct 30 03:07:22.821252 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:07:22.821231    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aff33fb_25cf_4fe3_ab90_8b684710191e.slice/crio-a1d5f1f0027658c02b0f251bff9f3a32418a1b444225d7ce71a480c320547c10 WatchSource:0}: Error finding container a1d5f1f0027658c02b0f251bff9f3a32418a1b444225d7ce71a480c320547c10: Status 404 returned error can't find the container with id a1d5f1f0027658c02b0f251bff9f3a32418a1b444225d7ce71a480c320547c10
Oct 30 03:07:23.113119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:23.112965    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"]
Oct 30 03:07:23.115341 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:07:23.115317    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45c6f06_e176_4950_9230_8da4d7c60037.slice/crio-46405f91a9e5c8b49eac73d08350550c87add03f0d5598d0efb40c9b8d51d465 WatchSource:0}: Error finding container 46405f91a9e5c8b49eac73d08350550c87add03f0d5598d0efb40c9b8d51d465: Status 404 returned error can't find the container with id 46405f91a9e5c8b49eac73d08350550c87add03f0d5598d0efb40c9b8d51d465
Oct 30 03:07:23.552789 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:23.552749    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" event={"ID":"b45c6f06-e176-4950-9230-8da4d7c60037","Type":"ContainerStarted","Data":"719a3bb4c26fd2ad9941df4f8fe736703d992798b8b972243c73593e0526d81f"}
Oct 30 03:07:23.553300 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:23.552828    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" event={"ID":"b45c6f06-e176-4950-9230-8da4d7c60037","Type":"ContainerStarted","Data":"46405f91a9e5c8b49eac73d08350550c87add03f0d5598d0efb40c9b8d51d465"}
Oct 30 03:07:23.554272 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:23.554245    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" event={"ID":"1aff33fb-25cf-4fe3-ab90-8b684710191e","Type":"ContainerStarted","Data":"426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e"}
Oct 30 03:07:23.554414 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:23.554301    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" event={"ID":"1aff33fb-25cf-4fe3-ab90-8b684710191e","Type":"ContainerStarted","Data":"a1d5f1f0027658c02b0f251bff9f3a32418a1b444225d7ce71a480c320547c10"}
Oct 30 03:07:25.921764 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:07:25.921727    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:07:26.644162 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:07:26.644131    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aff33fb_25cf_4fe3_ab90_8b684710191e.slice/crio-426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:07:27.573594 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:27.573564    4885 generic.go:356] "Generic (PLEG): container finished" podID="b45c6f06-e176-4950-9230-8da4d7c60037" containerID="719a3bb4c26fd2ad9941df4f8fe736703d992798b8b972243c73593e0526d81f" exitCode=0
Oct 30 03:07:27.574100 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:27.573640    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" event={"ID":"b45c6f06-e176-4950-9230-8da4d7c60037","Type":"ContainerDied","Data":"719a3bb4c26fd2ad9941df4f8fe736703d992798b8b972243c73593e0526d81f"}
Oct 30 03:07:27.575301 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:27.575281    4885 generic.go:356] "Generic (PLEG): container finished" podID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerID="426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e" exitCode=0
Oct 30 03:07:27.575365 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:27.575308    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" event={"ID":"1aff33fb-25cf-4fe3-ab90-8b684710191e","Type":"ContainerDied","Data":"426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e"}
Oct 30 03:07:39.679692 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:39.679648    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" event={"ID":"b45c6f06-e176-4950-9230-8da4d7c60037","Type":"ContainerStarted","Data":"2675f6b84caf48ee5f19ea8501380f80d021c14404db636848d731dea123480c"}
Oct 30 03:07:41.692828 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:41.692791    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" event={"ID":"b45c6f06-e176-4950-9230-8da4d7c60037","Type":"ContainerStarted","Data":"f44e25d2d5af424460958fb31f75d5e15a6b9f2c91cf34b006f0bf5506058232"}
Oct 30 03:07:41.693262 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:41.692864    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:41.693262 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:41.692893    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:41.694557 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:41.694517    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 03:07:41.745100 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:41.745025    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podStartSLOduration=2.205616599 podStartE2EDuration="19.745005615s" totalImagesPullingTime="13.949865967s" totalInitContainerRuntime="3.589523049s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:41.743430241 +0000 UTC m=+767.253599266" watchObservedRunningTime="2026-10-30 03:07:41.745005615 +0000 UTC m=+767.255174642"
Oct 30 03:07:42.698300 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:42.698258    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 03:07:45.714130 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:45.714098    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" event={"ID":"1aff33fb-25cf-4fe3-ab90-8b684710191e","Type":"ContainerStarted","Data":"f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7"}
Oct 30 03:07:45.714584 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:45.714160    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" event={"ID":"1aff33fb-25cf-4fe3-ab90-8b684710191e","Type":"ContainerStarted","Data":"ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720"}
Oct 30 03:07:47.703859 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:47.703837    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:07:47.704350 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:47.704328    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 03:07:48.738046 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:48.738007    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" event={"ID":"1aff33fb-25cf-4fe3-ab90-8b684710191e","Type":"ContainerStarted","Data":"19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f"}
Oct 30 03:07:48.738538 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:48.738423    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:48.738538 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:48.738478    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:48.738538 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:48.738512    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:48.740590 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:48.740560    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Oct 30 03:07:48.741353 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:48.741320    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:07:48.801701 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:48.801124    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podStartSLOduration=3.151636671 podStartE2EDuration="26.801108886s" totalImagesPullingTime="19.919413899s" totalInitContainerRuntime="3.730058316s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:48.790299608 +0000 UTC m=+774.300468633" watchObservedRunningTime="2026-10-30 03:07:48.801108886 +0000 UTC m=+774.311277917"
Oct 30 03:07:49.743132 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:49.743086    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Oct 30 03:07:49.743674 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:49.743647    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:07:54.747941 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:54.747911    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:07:54.748513 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:54.748490    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Oct 30 03:07:54.748695 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:54.748665    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:07:55.922519 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:07:55.922489    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:07:57.704411 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:07:57.704375    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 03:08:04.748390 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:04.748344    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Oct 30 03:08:04.748971 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:04.748731    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:08:07.704645 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:07.704591    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 03:08:14.748957 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:14.748916    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Oct 30 03:08:14.749515 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:14.749378    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:08:17.705179 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:17.705145    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 03:08:24.748494 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:24.748450    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Oct 30 03:08:24.749030 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:24.748942    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:08:25.922217 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:08:25.922177    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:08:27.704252 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:27.704210    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 03:08:34.749354 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:34.749305    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Oct 30 03:08:34.749856 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:34.749816    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:08:37.704847 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:37.704812    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 03:08:44.748381 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:44.748336    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Oct 30 03:08:44.748816 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:44.748783    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:08:47.704833 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:47.704792    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 03:08:54.749329 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:54.749275    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Oct 30 03:08:54.749856 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:54.749750    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:08:55.922731 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:08:55.922693    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:08:57.705537 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:08:57.704569    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 03:09:02.618899 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:02.618856    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Oct 30 03:09:04.749393 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:04.749367    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:09:04.749846 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:04.749738    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:09:12.623922 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:12.623829    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:09:17.909187 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:17.909149    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"]
Oct 30 03:09:17.909709 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:17.909623    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" containerID="cri-o://ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720" gracePeriod=30
Oct 30 03:09:17.909930 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:17.909881    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" containerID="cri-o://19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f" gracePeriod=30
Oct 30 03:09:17.910561 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:17.910115    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kube-rbac-proxy" containerID="cri-o://f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7" gracePeriod=30
Oct 30 03:09:18.130504 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:18.130465    4885 generic.go:356] "Generic (PLEG): container finished" podID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerID="f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7" exitCode=0
Oct 30 03:09:18.130766 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:18.130509    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" event={"ID":"1aff33fb-25cf-4fe3-ab90-8b684710191e","Type":"ContainerDied","Data":"f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7"}
Oct 30 03:09:19.743199 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:19.743161    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Oct 30 03:09:23.156024 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:23.155993    4885 generic.go:356] "Generic (PLEG): container finished" podID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerID="ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720" exitCode=0
Oct 30 03:09:23.156415 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:23.156068    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" event={"ID":"1aff33fb-25cf-4fe3-ab90-8b684710191e","Type":"ContainerDied","Data":"ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720"}
Oct 30 03:09:24.149483 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:24.149453    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"]
Oct 30 03:09:24.149729 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:24.149702    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container" containerID="cri-o://2675f6b84caf48ee5f19ea8501380f80d021c14404db636848d731dea123480c" gracePeriod=30
Oct 30 03:09:24.149815 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:24.149737    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kube-rbac-proxy" containerID="cri-o://f44e25d2d5af424460958fb31f75d5e15a6b9f2c91cf34b006f0bf5506058232" gracePeriod=30
Oct 30 03:09:24.743338 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:24.743302    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Oct 30 03:09:24.748648 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:24.748605    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Oct 30 03:09:24.749028 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:24.749010    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:25.167178 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:25.167094    4885 generic.go:356] "Generic (PLEG): container finished" podID="b45c6f06-e176-4950-9230-8da4d7c60037" containerID="f44e25d2d5af424460958fb31f75d5e15a6b9f2c91cf34b006f0bf5506058232" exitCode=0
Oct 30 03:09:25.167178 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:25.167165    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" event={"ID":"b45c6f06-e176-4950-9230-8da4d7c60037","Type":"ContainerDied","Data":"f44e25d2d5af424460958fb31f75d5e15a6b9f2c91cf34b006f0bf5506058232"}
Oct 30 03:09:25.921900 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:09:25.921865    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:09:27.699015 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:27.698974    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Oct 30 03:09:28.183658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.183607    4885 generic.go:356] "Generic (PLEG): container finished" podID="b45c6f06-e176-4950-9230-8da4d7c60037" containerID="2675f6b84caf48ee5f19ea8501380f80d021c14404db636848d731dea123480c" exitCode=0
Oct 30 03:09:28.183658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.183647    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" event={"ID":"b45c6f06-e176-4950-9230-8da4d7c60037","Type":"ContainerDied","Data":"2675f6b84caf48ee5f19ea8501380f80d021c14404db636848d731dea123480c"}
Oct 30 03:09:28.183796 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.183703    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5" event={"ID":"b45c6f06-e176-4950-9230-8da4d7c60037","Type":"ContainerDied","Data":"46405f91a9e5c8b49eac73d08350550c87add03f0d5598d0efb40c9b8d51d465"}
Oct 30 03:09:28.183796 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.183726    4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46405f91a9e5c8b49eac73d08350550c87add03f0d5598d0efb40c9b8d51d465"
Oct 30 03:09:28.191785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.191767    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:09:28.267833 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.267804    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-cabundle-cert\") pod \"b45c6f06-e176-4950-9230-8da4d7c60037\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") "
Oct 30 03:09:28.267983 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.267880    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-kube-rbac-proxy-sar-config\") pod \"b45c6f06-e176-4950-9230-8da4d7c60037\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") "
Oct 30 03:09:28.267983 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.267920    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b45c6f06-e176-4950-9230-8da4d7c60037-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b45c6f06-e176-4950-9230-8da4d7c60037-proxy-tls\") pod \"b45c6f06-e176-4950-9230-8da4d7c60037\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") "
Oct 30 03:09:28.267983 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.267939    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45c6f06-e176-4950-9230-8da4d7c60037-kube-api-access-pf248\" (UniqueName: \"kubernetes.io/projected/b45c6f06-e176-4950-9230-8da4d7c60037-kube-api-access-pf248\") pod \"b45c6f06-e176-4950-9230-8da4d7c60037\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") "
Oct 30 03:09:28.267983 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.267971    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45c6f06-e176-4950-9230-8da4d7c60037-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b45c6f06-e176-4950-9230-8da4d7c60037-kserve-provision-location\") pod \"b45c6f06-e176-4950-9230-8da4d7c60037\" (UID: \"b45c6f06-e176-4950-9230-8da4d7c60037\") "
Oct 30 03:09:28.268223 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.268170    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-cabundle-cert" pod "b45c6f06-e176-4950-9230-8da4d7c60037" (UID: "b45c6f06-e176-4950-9230-8da4d7c60037"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:28.268357 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.268331    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-kube-rbac-proxy-sar-config" pod "b45c6f06-e176-4950-9230-8da4d7c60037" (UID: "b45c6f06-e176-4950-9230-8da4d7c60037"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:28.268357 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.268338    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45c6f06-e176-4950-9230-8da4d7c60037-kserve-provision-location" pod "b45c6f06-e176-4950-9230-8da4d7c60037" (UID: "b45c6f06-e176-4950-9230-8da4d7c60037"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:28.270285 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.270262    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45c6f06-e176-4950-9230-8da4d7c60037-kube-api-access-pf248" pod "b45c6f06-e176-4950-9230-8da4d7c60037" (UID: "b45c6f06-e176-4950-9230-8da4d7c60037"). InnerVolumeSpecName "kube-api-access-pf248". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:28.270348 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.270299    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45c6f06-e176-4950-9230-8da4d7c60037-proxy-tls" pod "b45c6f06-e176-4950-9230-8da4d7c60037" (UID: "b45c6f06-e176-4950-9230-8da4d7c60037"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:09:28.369078 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.369016    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:09:28.369078 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.369038    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b45c6f06-e176-4950-9230-8da4d7c60037-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:09:28.369078 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.369048    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf248\" (UniqueName: \"kubernetes.io/projected/b45c6f06-e176-4950-9230-8da4d7c60037-kube-api-access-pf248\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:09:28.369078 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.369056    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b45c6f06-e176-4950-9230-8da4d7c60037-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:09:28.369078 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:28.369068    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b45c6f06-e176-4950-9230-8da4d7c60037-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:09:29.187526 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:29.187497    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"
Oct 30 03:09:29.248128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:29.248095    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"]
Oct 30 03:09:29.258560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:29.258537    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-v2-kserve-predictor-5df974dc5-jfdt5"]
Oct 30 03:09:29.744060 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:29.744020    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Oct 30 03:09:29.744239 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:29.744142    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:09:30.627019 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:30.626985    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" path="/var/lib/kubelet/pods/b45c6f06-e176-4950-9230-8da4d7c60037/volumes"
Oct 30 03:09:31.876973 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.876944    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"]
Oct 30 03:09:31.877378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.877331    4885 cpu_manager.go:414] "containerMap: removing container" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container"
Oct 30 03:09:31.877378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.877349    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container"
Oct 30 03:09:31.877378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.877377    4885 cpu_manager.go:414] "containerMap: removing container" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="storage-initializer"
Oct 30 03:09:31.877486 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.877385    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="storage-initializer"
Oct 30 03:09:31.877486 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.877405    4885 cpu_manager.go:414] "containerMap: removing container" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kube-rbac-proxy"
Oct 30 03:09:31.877486 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.877414    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kube-rbac-proxy"
Oct 30 03:09:31.877585 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.877509    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kserve-container"
Oct 30 03:09:31.877585 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.877524    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45c6f06-e176-4950-9230-8da4d7c60037" containerName="kube-rbac-proxy"
Oct 30 03:09:31.881914 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.881895    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:31.907737 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.907712    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:09:31.907737 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.907720    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:09:31.907951 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.907722    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:09:31.907951 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.907923    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-primary-42d50b-kube-rbac-proxy-sar-config\""
Oct 30 03:09:31.907951 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.907936    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:09:31.907951 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.907936    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-primary-42d50b-predictor-serving-cert\""
Oct 30 03:09:31.908141 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.907981    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:09:31.909518 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.909502    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"]
Oct 30 03:09:31.998416 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.998391    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-rbac-proxy-sar-config\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:31.998547 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.998441    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-proxy-tls\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:31.998547 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.998487    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kserve-provision-location\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:31.998654 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.998564    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-cabundle-cert\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:31.998654 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:31.998603    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjjs5\" (UniqueName: \"kubernetes.io/projected/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-api-access-vjjs5\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:32.099732 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:32.099702    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-proxy-tls\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:32.099869 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:32.099737    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kserve-provision-location\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:32.099869 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:32.099764    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-cabundle-cert\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:32.099869 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:32.099789    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjjs5\" (UniqueName: \"kubernetes.io/projected/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-api-access-vjjs5\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:32.099869 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:32.099840    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-rbac-proxy-sar-config\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:32.100142 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:32.100120    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kserve-provision-location\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:32.100570 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:32.100549    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-cabundle-cert\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:32.100643 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:32.100549    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-rbac-proxy-sar-config\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:32.102355 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:32.102335    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-proxy-tls\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:32.121471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:32.121452    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjjs5\" (UniqueName: \"kubernetes.io/projected/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-api-access-vjjs5\") pod \"isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") " pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:32.201358 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:32.201295    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:32.504677 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:32.504561    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"]
Oct 30 03:09:32.506986 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:09:32.506958    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21b1af3_c1c2_4990_bb51_37b5efc00dfe.slice/crio-5167674a06b9494b47913ab846f008cd813b06c614622fc0875bb0746a735438 WatchSource:0}: Error finding container 5167674a06b9494b47913ab846f008cd813b06c614622fc0875bb0746a735438: Status 404 returned error can't find the container with id 5167674a06b9494b47913ab846f008cd813b06c614622fc0875bb0746a735438
Oct 30 03:09:33.205936 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:33.205892    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" event={"ID":"a21b1af3-c1c2-4990-bb51-37b5efc00dfe","Type":"ContainerStarted","Data":"f2d96ab74fe2ef09d257bad8c5a643763326167c5c7914c0fe22e94c07118e57"}
Oct 30 03:09:33.206374 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:33.205963    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" event={"ID":"a21b1af3-c1c2-4990-bb51-37b5efc00dfe","Type":"ContainerStarted","Data":"5167674a06b9494b47913ab846f008cd813b06c614622fc0875bb0746a735438"}
Oct 30 03:09:34.743847 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:34.743802    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Oct 30 03:09:34.749314 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:34.749283    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Oct 30 03:09:34.749658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:34.749599    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:37.224219 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:37.224185    4885 generic.go:356] "Generic (PLEG): container finished" podID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerID="f2d96ab74fe2ef09d257bad8c5a643763326167c5c7914c0fe22e94c07118e57" exitCode=0
Oct 30 03:09:37.224604 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:37.224263    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" event={"ID":"a21b1af3-c1c2-4990-bb51-37b5efc00dfe","Type":"ContainerDied","Data":"f2d96ab74fe2ef09d257bad8c5a643763326167c5c7914c0fe22e94c07118e57"}
Oct 30 03:09:38.231381 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:38.231347    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" event={"ID":"a21b1af3-c1c2-4990-bb51-37b5efc00dfe","Type":"ContainerStarted","Data":"f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626"}
Oct 30 03:09:38.231793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:38.231438    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" event={"ID":"a21b1af3-c1c2-4990-bb51-37b5efc00dfe","Type":"ContainerStarted","Data":"75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d"}
Oct 30 03:09:38.231793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:38.231759    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:38.284178 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:38.284129    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podStartSLOduration=3.604932293 podStartE2EDuration="7.284112845s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.679180552s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:38.282485954 +0000 UTC m=+883.792654989" watchObservedRunningTime="2026-10-30 03:09:38.284112845 +0000 UTC m=+883.794281874"
Oct 30 03:09:39.236350 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:39.236310    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:39.237763 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:39.237737    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Oct 30 03:09:39.743501 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:39.743462    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Oct 30 03:09:40.240419 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:40.240376    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Oct 30 03:09:44.744029 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:44.743983    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Oct 30 03:09:44.748397 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:44.748361    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Oct 30 03:09:44.748546 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:44.748530    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:09:44.748680 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:44.748655    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:44.748775 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:44.748761    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:09:45.244373 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:45.244345    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:09:45.244925 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:45.244897    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Oct 30 03:09:48.077686 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.077662    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:09:48.232248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.232216    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-api-access-mwsdl\" (UniqueName: \"kubernetes.io/projected/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-api-access-mwsdl\") pod \"1aff33fb-25cf-4fe3-ab90-8b684710191e\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") "
Oct 30 03:09:48.232248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.232250    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aff33fb-25cf-4fe3-ab90-8b684710191e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1aff33fb-25cf-4fe3-ab90-8b684710191e-proxy-tls\") pod \"1aff33fb-25cf-4fe3-ab90-8b684710191e\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") "
Oct 30 03:09:48.232498 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.232285    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aff33fb-25cf-4fe3-ab90-8b684710191e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1aff33fb-25cf-4fe3-ab90-8b684710191e-kserve-provision-location\") pod \"1aff33fb-25cf-4fe3-ab90-8b684710191e\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") "
Oct 30 03:09:48.232498 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.232349    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-cabundle-cert\") pod \"1aff33fb-25cf-4fe3-ab90-8b684710191e\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") "
Oct 30 03:09:48.232498 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.232382    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-rbac-proxy-sar-config\") pod \"1aff33fb-25cf-4fe3-ab90-8b684710191e\" (UID: \"1aff33fb-25cf-4fe3-ab90-8b684710191e\") "
Oct 30 03:09:48.232777 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.232689    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aff33fb-25cf-4fe3-ab90-8b684710191e-kserve-provision-location" pod "1aff33fb-25cf-4fe3-ab90-8b684710191e" (UID: "1aff33fb-25cf-4fe3-ab90-8b684710191e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:48.232844 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.232731    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-cabundle-cert" pod "1aff33fb-25cf-4fe3-ab90-8b684710191e" (UID: "1aff33fb-25cf-4fe3-ab90-8b684710191e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:48.232901 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.232858    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-rbac-proxy-sar-config" pod "1aff33fb-25cf-4fe3-ab90-8b684710191e" (UID: "1aff33fb-25cf-4fe3-ab90-8b684710191e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:48.234635 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.234593    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aff33fb-25cf-4fe3-ab90-8b684710191e-proxy-tls" pod "1aff33fb-25cf-4fe3-ab90-8b684710191e" (UID: "1aff33fb-25cf-4fe3-ab90-8b684710191e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:09:48.234750 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.234665    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-api-access-mwsdl" pod "1aff33fb-25cf-4fe3-ab90-8b684710191e" (UID: "1aff33fb-25cf-4fe3-ab90-8b684710191e"). InnerVolumeSpecName "kube-api-access-mwsdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:48.276772 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.276744    4885 generic.go:356] "Generic (PLEG): container finished" podID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerID="19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f" exitCode=0
Oct 30 03:09:48.276876 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.276818    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"
Oct 30 03:09:48.276876 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.276827    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" event={"ID":"1aff33fb-25cf-4fe3-ab90-8b684710191e","Type":"ContainerDied","Data":"19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f"}
Oct 30 03:09:48.276942 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.276894    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg" event={"ID":"1aff33fb-25cf-4fe3-ab90-8b684710191e","Type":"ContainerDied","Data":"a1d5f1f0027658c02b0f251bff9f3a32418a1b444225d7ce71a480c320547c10"}
Oct 30 03:09:48.276942 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.276928    4885 scope.go:122] "RemoveContainer" containerID="19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f"
Oct 30 03:09:48.286437 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.286365    4885 scope.go:122] "RemoveContainer" containerID="f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7"
Oct 30 03:09:48.295680 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.295664    4885 scope.go:122] "RemoveContainer" containerID="ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720"
Oct 30 03:09:48.302845 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.302829    4885 scope.go:122] "RemoveContainer" containerID="426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e"
Oct 30 03:09:48.310078 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.310062    4885 scope.go:122] "RemoveContainer" containerID="19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f"
Oct 30 03:09:48.310359 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:09:48.310340    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f\": container with ID starting with 19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f not found: ID does not exist" containerID="19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f"
Oct 30 03:09:48.310432 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.310365    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f"} err="failed to get container status \"19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f\": rpc error: code = NotFound desc = could not find container \"19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f\": container with ID starting with 19b8220b95c1e5d855dd03d17f099f6f77a77edf2960d93af4ca13194122db7f not found: ID does not exist"
Oct 30 03:09:48.310432 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.310382    4885 scope.go:122] "RemoveContainer" containerID="f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7"
Oct 30 03:09:48.310627 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:09:48.310598    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7\": container with ID starting with f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7 not found: ID does not exist" containerID="f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7"
Oct 30 03:09:48.310673 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.310636    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7"} err="failed to get container status \"f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7\": rpc error: code = NotFound desc = could not find container \"f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7\": container with ID starting with f8302be770a64e04186debf92615352085dec4b667835715e37a0f366be2d0e7 not found: ID does not exist"
Oct 30 03:09:48.310673 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.310655    4885 scope.go:122] "RemoveContainer" containerID="ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720"
Oct 30 03:09:48.310906 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:09:48.310888    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720\": container with ID starting with ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720 not found: ID does not exist" containerID="ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720"
Oct 30 03:09:48.310947 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.310911    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720"} err="failed to get container status \"ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720\": rpc error: code = NotFound desc = could not find container \"ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720\": container with ID starting with ff6f8d6f252ab161f98ddd9d90b6fc6a766b11a74b569a5e0131ea0fa6de8720 not found: ID does not exist"
Oct 30 03:09:48.310947 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.310923    4885 scope.go:122] "RemoveContainer" containerID="426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e"
Oct 30 03:09:48.311149 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:09:48.311131    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e\": container with ID starting with 426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e not found: ID does not exist" containerID="426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e"
Oct 30 03:09:48.311189 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.311154    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e"} err="failed to get container status \"426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e\": rpc error: code = NotFound desc = could not find container \"426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e\": container with ID starting with 426855e6cb8c0667f5e9604d91712d6513efd012238026a3eb12831b7aee2c2e not found: ID does not exist"
Oct 30 03:09:48.325806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.325788    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"]
Oct 30 03:09:48.329655 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.329638    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-batcher-predictor-857545b8c4-qk6mg"]
Oct 30 03:09:48.333747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.333725    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1aff33fb-25cf-4fe3-ab90-8b684710191e-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:09:48.333747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.333744    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:09:48.333900 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.333756    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:09:48.333900 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.333765    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwsdl\" (UniqueName: \"kubernetes.io/projected/1aff33fb-25cf-4fe3-ab90-8b684710191e-kube-api-access-mwsdl\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:09:48.333900 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.333776    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1aff33fb-25cf-4fe3-ab90-8b684710191e-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:09:48.624085 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:48.624008    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" path="/var/lib/kubelet/pods/1aff33fb-25cf-4fe3-ab90-8b684710191e/volumes"
Oct 30 03:09:49.762807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.762765    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"]
Oct 30 03:09:49.763277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.763232    4885 cpu_manager.go:414] "containerMap: removing container" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent"
Oct 30 03:09:49.763277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.763255    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent"
Oct 30 03:09:49.763393 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.763284    4885 cpu_manager.go:414] "containerMap: removing container" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kube-rbac-proxy"
Oct 30 03:09:49.763393 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.763292    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kube-rbac-proxy"
Oct 30 03:09:49.763393 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.763313    4885 cpu_manager.go:414] "containerMap: removing container" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="storage-initializer"
Oct 30 03:09:49.763393 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.763321    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="storage-initializer"
Oct 30 03:09:49.763393 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.763340    4885 cpu_manager.go:414] "containerMap: removing container" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container"
Oct 30 03:09:49.763393 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.763347    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container"
Oct 30 03:09:49.763721 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.763488    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="agent"
Oct 30 03:09:49.763721 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.763506    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kube-rbac-proxy"
Oct 30 03:09:49.763721 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.763518    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aff33fb-25cf-4fe3-ab90-8b684710191e" containerName="kserve-container"
Oct 30 03:09:49.767094 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.767073    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.779946 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.779925    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Oct 30 03:09:49.789966 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.789948    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:09:49.790099 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.789966    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:09:49.790099 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.790015    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Oct 30 03:09:49.790099 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.790053    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:09:49.791874 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.791846    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"]
Oct 30 03:09:49.847034 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.847002    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.847177 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.847055    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d6783b1-7e11-4004-8c72-742837cb58b7-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.847177 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.847138    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.847251 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.847183    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d6783b1-7e11-4004-8c72-742837cb58b7-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.847285 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.847244    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdtv\" (UniqueName: \"kubernetes.io/projected/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-api-access-2jdtv\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.880180 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.880155    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:09:49.880180 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.880155    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:09:49.948329 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.948299    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d6783b1-7e11-4004-8c72-742837cb58b7-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.948477 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.948336    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.948477 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.948368    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d6783b1-7e11-4004-8c72-742837cb58b7-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.948477 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.948399    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdtv\" (UniqueName: \"kubernetes.io/projected/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-api-access-2jdtv\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.948477 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.948442    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.948907 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.948883    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d6783b1-7e11-4004-8c72-742837cb58b7-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.949095 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.949078    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.949132 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.949105    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.951037 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.951020    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d6783b1-7e11-4004-8c72-742837cb58b7-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:49.970002 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:49.969977    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdtv\" (UniqueName: \"kubernetes.io/projected/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-api-access-2jdtv\") pod \"isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:50.086117 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:50.086038    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:50.394090 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:50.394013    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"]
Oct 30 03:09:50.397126 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:09:50.397095    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6783b1_7e11_4004_8c72_742837cb58b7.slice/crio-8128d0b7022f8093d319ee66b8055ae1d2fb662261ea21983a24d4397b3610c9 WatchSource:0}: Error finding container 8128d0b7022f8093d319ee66b8055ae1d2fb662261ea21983a24d4397b3610c9: Status 404 returned error can't find the container with id 8128d0b7022f8093d319ee66b8055ae1d2fb662261ea21983a24d4397b3610c9
Oct 30 03:09:51.292269 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:51.292234    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" event={"ID":"9d6783b1-7e11-4004-8c72-742837cb58b7","Type":"ContainerStarted","Data":"4d4a2ea7eaff1c2e3e069ac4a2485d5b9e7283a9f3c0bb3e5908a18cac84605d"}
Oct 30 03:09:51.292687 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:51.292311    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" event={"ID":"9d6783b1-7e11-4004-8c72-742837cb58b7","Type":"ContainerStarted","Data":"8128d0b7022f8093d319ee66b8055ae1d2fb662261ea21983a24d4397b3610c9"}
Oct 30 03:09:54.306922 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:54.306887    4885 generic.go:356] "Generic (PLEG): container finished" podID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerID="4d4a2ea7eaff1c2e3e069ac4a2485d5b9e7283a9f3c0bb3e5908a18cac84605d" exitCode=0
Oct 30 03:09:54.307383 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:54.306962    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" event={"ID":"9d6783b1-7e11-4004-8c72-742837cb58b7","Type":"ContainerDied","Data":"4d4a2ea7eaff1c2e3e069ac4a2485d5b9e7283a9f3c0bb3e5908a18cac84605d"}
Oct 30 03:09:54.686738 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:54.686652    4885 scope.go:122] "RemoveContainer" containerID="c27806274aeac24f41e0ce816fc7ba90e67f5794e7b271dd939a63f0a9ac4a1f"
Oct 30 03:09:54.702280 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:54.702200    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:09:54.702757 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:54.702734    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:09:54.708322 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:54.708301    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:09:54.708852 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:54.708830    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:09:55.245460 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:55.245418    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Oct 30 03:09:55.314448 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:55.314417    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" event={"ID":"9d6783b1-7e11-4004-8c72-742837cb58b7","Type":"ContainerStarted","Data":"b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998"}
Oct 30 03:09:55.314862 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:55.314515    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" event={"ID":"9d6783b1-7e11-4004-8c72-742837cb58b7","Type":"ContainerStarted","Data":"d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a"}
Oct 30 03:09:55.314862 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:55.314541    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" event={"ID":"9d6783b1-7e11-4004-8c72-742837cb58b7","Type":"ContainerStarted","Data":"5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f"}
Oct 30 03:09:55.314942 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:55.314860    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:55.315036 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:55.315017    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:55.316511 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:55.316483    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:09:55.378221 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:55.378164    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podStartSLOduration=2.665328021 podStartE2EDuration="6.378146627s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.712818606s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:55.367581579 +0000 UTC m=+900.877750626" watchObservedRunningTime="2026-10-30 03:09:55.378146627 +0000 UTC m=+900.888315655"
Oct 30 03:09:55.922127 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:09:55.922092    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:09:56.318817 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:56.318768    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:09:56.319296 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:56.318796    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:09:56.320122 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:56.320097    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:57.322913 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:57.322869    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:09:57.323352 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:09:57.323250    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:10:02.327583 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:02.327554    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:10:02.328115 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:02.328088    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:10:02.328433 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:02.328410    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:10:05.245774 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:05.245737    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Oct 30 03:10:12.328212 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:12.328171    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:10:12.328677 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:12.328652    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:10:15.245862 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:15.245822    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Oct 30 03:10:22.328558 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:22.328523    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:10:22.329079 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:22.329000    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:10:25.245587 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:25.245547    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Oct 30 03:10:25.921495 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:10:25.921459    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:10:32.328125 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:32.328076    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:10:32.328602 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:32.328581    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:10:35.245216 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:35.245172    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Oct 30 03:10:42.328579 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:42.328537    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:10:42.329114 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:42.329091    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:10:45.245702 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:45.245663    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Oct 30 03:10:52.328506 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:52.328464    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:10:52.329070 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:52.329048    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:10:55.245518 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:10:55.245486    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:10:55.921976 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:10:55.921944    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:11:02.328186 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:02.328125    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:11:02.328629 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:02.328578    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:11:12.329520 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:12.329488    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:11:12.329991 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:12.329729    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:11:17.356185 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.356152    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"]
Oct 30 03:11:17.360325 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.360306    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.374647 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.374627    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-secondary-42d50b-kube-rbac-proxy-sar-config\""
Oct 30 03:11:17.384987 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.384964    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"invalid-s3-secret-42d50b\""
Oct 30 03:11:17.385180 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.385158    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-secondary-42d50b-predictor-serving-cert\""
Oct 30 03:11:17.385370 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.385350    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"invalid-s3-sa-42d50b-dockercfg-vps8r\""
Oct 30 03:11:17.387140 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.387116    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"]
Oct 30 03:11:17.467512 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.467479    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-cabundle-cert\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.467676 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.467521    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17ff5f4b-d5c9-4187-a283-da813d6419fc-proxy-tls\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.467676 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.467544    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17ff5f4b-d5c9-4187-a283-da813d6419fc-kserve-provision-location\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.467761 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.467727    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnjgp\" (UniqueName: \"kubernetes.io/projected/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-api-access-xnjgp\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.467799 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.467781    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.568918 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.568885    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-cabundle-cert\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.568918 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.568922    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17ff5f4b-d5c9-4187-a283-da813d6419fc-proxy-tls\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.569158 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.568944    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17ff5f4b-d5c9-4187-a283-da813d6419fc-kserve-provision-location\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.569158 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.569023    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnjgp\" (UniqueName: \"kubernetes.io/projected/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-api-access-xnjgp\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.569158 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.569048    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.569443 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.569421    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17ff5f4b-d5c9-4187-a283-da813d6419fc-kserve-provision-location\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.569662 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.569641    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-cabundle-cert\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.569720 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.569689    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.571707 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.571683    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17ff5f4b-d5c9-4187-a283-da813d6419fc-proxy-tls\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.590098 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.590077    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnjgp\" (UniqueName: \"kubernetes.io/projected/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-api-access-xnjgp\") pod \"isvc-secondary-42d50b-predictor-7d99dff557-78s7x\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") " pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.677198 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.677123    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:17.981998 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:17.981802    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"]
Oct 30 03:11:17.984191 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:11:17.984164    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ff5f4b_d5c9_4187_a283_da813d6419fc.slice/crio-84527267bc4009d081b876db1769df0732446d7dd5e7ed9f24a1203b9166fcb9 WatchSource:0}: Error finding container 84527267bc4009d081b876db1769df0732446d7dd5e7ed9f24a1203b9166fcb9: Status 404 returned error can't find the container with id 84527267bc4009d081b876db1769df0732446d7dd5e7ed9f24a1203b9166fcb9
Oct 30 03:11:18.683159 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:18.683117    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x" event={"ID":"17ff5f4b-d5c9-4187-a283-da813d6419fc","Type":"ContainerStarted","Data":"ce2c61d309b3ed26f48e5c871428ec06b1653253ee98c0d4393cf9136cfac840"}
Oct 30 03:11:18.683565 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:18.683194    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x" event={"ID":"17ff5f4b-d5c9-4187-a283-da813d6419fc","Type":"ContainerStarted","Data":"84527267bc4009d081b876db1769df0732446d7dd5e7ed9f24a1203b9166fcb9"}
Oct 30 03:11:21.697634 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:21.697591    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-secondary-42d50b-predictor-7d99dff557-78s7x_17ff5f4b-d5c9-4187-a283-da813d6419fc/storage-initializer/0.log"
Oct 30 03:11:21.698008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:21.697648    4885 generic.go:356] "Generic (PLEG): container finished" podID="17ff5f4b-d5c9-4187-a283-da813d6419fc" containerID="ce2c61d309b3ed26f48e5c871428ec06b1653253ee98c0d4393cf9136cfac840" exitCode=1
Oct 30 03:11:21.698008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:21.697725    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x" event={"ID":"17ff5f4b-d5c9-4187-a283-da813d6419fc","Type":"ContainerDied","Data":"ce2c61d309b3ed26f48e5c871428ec06b1653253ee98c0d4393cf9136cfac840"}
Oct 30 03:11:22.703189 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:22.703157    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-secondary-42d50b-predictor-7d99dff557-78s7x_17ff5f4b-d5c9-4187-a283-da813d6419fc/storage-initializer/0.log"
Oct 30 03:11:22.703597 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:22.703250    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x" event={"ID":"17ff5f4b-d5c9-4187-a283-da813d6419fc","Type":"ContainerStarted","Data":"28cf5a44d077d1f4ab4577114a65978d724b95180c41f662ed067c015946a806"}
Oct 30 03:11:24.714017 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:24.713992    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-secondary-42d50b-predictor-7d99dff557-78s7x_17ff5f4b-d5c9-4187-a283-da813d6419fc/storage-initializer/1.log"
Oct 30 03:11:24.714427 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:24.714412    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-secondary-42d50b-predictor-7d99dff557-78s7x_17ff5f4b-d5c9-4187-a283-da813d6419fc/storage-initializer/0.log"
Oct 30 03:11:24.714476 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:24.714444    4885 generic.go:356] "Generic (PLEG): container finished" podID="17ff5f4b-d5c9-4187-a283-da813d6419fc" containerID="28cf5a44d077d1f4ab4577114a65978d724b95180c41f662ed067c015946a806" exitCode=1
Oct 30 03:11:24.714509 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:24.714478    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x" event={"ID":"17ff5f4b-d5c9-4187-a283-da813d6419fc","Type":"ContainerDied","Data":"28cf5a44d077d1f4ab4577114a65978d724b95180c41f662ed067c015946a806"}
Oct 30 03:11:24.714539 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:24.714527    4885 scope.go:122] "RemoveContainer" containerID="ce2c61d309b3ed26f48e5c871428ec06b1653253ee98c0d4393cf9136cfac840"
Oct 30 03:11:24.714905 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:24.714888    4885 scope.go:122] "RemoveContainer" containerID="ce2c61d309b3ed26f48e5c871428ec06b1653253ee98c0d4393cf9136cfac840"
Oct 30 03:11:24.725778 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:24.725756    4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-42d50b-predictor-7d99dff557-78s7x_e2e-gw1-773c71d3d4(17ff5f4b-d5c9-4187-a283-da813d6419fc)\"" pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x" podUID="17ff5f4b-d5c9-4187-a283-da813d6419fc"
Oct 30 03:11:24.726318 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:24.726298    4885 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-42d50b-predictor-7d99dff557-78s7x_e2e-gw1-773c71d3d4_17ff5f4b-d5c9-4187-a283-da813d6419fc_0 in pod sandbox 84527267bc4009d081b876db1769df0732446d7dd5e7ed9f24a1203b9166fcb9 from index: no such id: 'ce2c61d309b3ed26f48e5c871428ec06b1653253ee98c0d4393cf9136cfac840'" containerID="ce2c61d309b3ed26f48e5c871428ec06b1653253ee98c0d4393cf9136cfac840"
Oct 30 03:11:24.726367 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:24.726327    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2c61d309b3ed26f48e5c871428ec06b1653253ee98c0d4393cf9136cfac840"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-42d50b-predictor-7d99dff557-78s7x_e2e-gw1-773c71d3d4_17ff5f4b-d5c9-4187-a283-da813d6419fc_0 in pod sandbox 84527267bc4009d081b876db1769df0732446d7dd5e7ed9f24a1203b9166fcb9 from index: no such id: 'ce2c61d309b3ed26f48e5c871428ec06b1653253ee98c0d4393cf9136cfac840'"
Oct 30 03:11:25.412127 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:25.412092    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"]
Oct 30 03:11:25.412474 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:25.412432    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" containerID="cri-o://5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f" gracePeriod=30
Oct 30 03:11:25.412653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:25.412495    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" containerID="cri-o://b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998" gracePeriod=30
Oct 30 03:11:25.412653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:25.412509    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kube-rbac-proxy" containerID="cri-o://d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a" gracePeriod=30
Oct 30 03:11:25.721900 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:25.721867    4885 generic.go:356] "Generic (PLEG): container finished" podID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerID="d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a" exitCode=0
Oct 30 03:11:25.722293 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:25.721940    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" event={"ID":"9d6783b1-7e11-4004-8c72-742837cb58b7","Type":"ContainerDied","Data":"d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a"}
Oct 30 03:11:25.723548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:25.723530    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-secondary-42d50b-predictor-7d99dff557-78s7x_17ff5f4b-d5c9-4187-a283-da813d6419fc/storage-initializer/1.log"
Oct 30 03:11:25.922563 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:25.922516    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:11:27.323392 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:27.323350    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Oct 30 03:11:30.748444 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:30.748414    4885 generic.go:356] "Generic (PLEG): container finished" podID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerID="5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f" exitCode=0
Oct 30 03:11:30.748859 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:30.748492    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" event={"ID":"9d6783b1-7e11-4004-8c72-742837cb58b7","Type":"ContainerDied","Data":"5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f"}
Oct 30 03:11:32.323834 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:32.323792    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Oct 30 03:11:32.328087 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:32.328061    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:11:32.328388 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:32.328368    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:11:33.386723 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.386692    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"]
Oct 30 03:11:33.485126 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.485087    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"]
Oct 30 03:11:33.485703 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.485423    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container" containerID="cri-o://75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d" gracePeriod=30
Oct 30 03:11:33.485703 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.485506    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kube-rbac-proxy" containerID="cri-o://f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626" gracePeriod=30
Oct 30 03:11:33.606728 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.606708    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-secondary-42d50b-predictor-7d99dff557-78s7x_17ff5f4b-d5c9-4187-a283-da813d6419fc/storage-initializer/1.log"
Oct 30 03:11:33.608757 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.608742    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:33.719076 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.719054    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ff5f4b-d5c9-4187-a283-da813d6419fc-proxy-tls\" (UniqueName: \"kubernetes.io/secret/17ff5f4b-d5c9-4187-a283-da813d6419fc-proxy-tls\") pod \"17ff5f4b-d5c9-4187-a283-da813d6419fc\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") "
Oct 30 03:11:33.719207 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.719100    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ff5f4b-d5c9-4187-a283-da813d6419fc-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17ff5f4b-d5c9-4187-a283-da813d6419fc-kserve-provision-location\") pod \"17ff5f4b-d5c9-4187-a283-da813d6419fc\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") "
Oct 30 03:11:33.719207 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.719140    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-rbac-proxy-sar-config\") pod \"17ff5f4b-d5c9-4187-a283-da813d6419fc\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") "
Oct 30 03:11:33.719207 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.719162    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-api-access-xnjgp\" (UniqueName: \"kubernetes.io/projected/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-api-access-xnjgp\") pod \"17ff5f4b-d5c9-4187-a283-da813d6419fc\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") "
Oct 30 03:11:33.719207 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.719192    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-cabundle-cert\") pod \"17ff5f4b-d5c9-4187-a283-da813d6419fc\" (UID: \"17ff5f4b-d5c9-4187-a283-da813d6419fc\") "
Oct 30 03:11:33.719516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.719452    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ff5f4b-d5c9-4187-a283-da813d6419fc-kserve-provision-location" pod "17ff5f4b-d5c9-4187-a283-da813d6419fc" (UID: "17ff5f4b-d5c9-4187-a283-da813d6419fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:33.719592 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.719521    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-rbac-proxy-sar-config" pod "17ff5f4b-d5c9-4187-a283-da813d6419fc" (UID: "17ff5f4b-d5c9-4187-a283-da813d6419fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:33.719671 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.719638    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-cabundle-cert" pod "17ff5f4b-d5c9-4187-a283-da813d6419fc" (UID: "17ff5f4b-d5c9-4187-a283-da813d6419fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:33.721244 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.721226    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-api-access-xnjgp" pod "17ff5f4b-d5c9-4187-a283-da813d6419fc" (UID: "17ff5f4b-d5c9-4187-a283-da813d6419fc"). InnerVolumeSpecName "kube-api-access-xnjgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:33.721414 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.721395    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ff5f4b-d5c9-4187-a283-da813d6419fc-proxy-tls" pod "17ff5f4b-d5c9-4187-a283-da813d6419fc" (UID: "17ff5f4b-d5c9-4187-a283-da813d6419fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:11:33.763807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.763783    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-secondary-42d50b-predictor-7d99dff557-78s7x_17ff5f4b-d5c9-4187-a283-da813d6419fc/storage-initializer/1.log"
Oct 30 03:11:33.763925 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.763879    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"
Oct 30 03:11:33.763925 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.763901    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x" event={"ID":"17ff5f4b-d5c9-4187-a283-da813d6419fc","Type":"ContainerDied","Data":"84527267bc4009d081b876db1769df0732446d7dd5e7ed9f24a1203b9166fcb9"}
Oct 30 03:11:33.764045 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.763977    4885 scope.go:122] "RemoveContainer" containerID="28cf5a44d077d1f4ab4577114a65978d724b95180c41f662ed067c015946a806"
Oct 30 03:11:33.766314 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.766292    4885 generic.go:356] "Generic (PLEG): container finished" podID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerID="f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626" exitCode=0
Oct 30 03:11:33.766419 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.766366    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" event={"ID":"a21b1af3-c1c2-4990-bb51-37b5efc00dfe","Type":"ContainerDied","Data":"f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626"}
Oct 30 03:11:33.820178 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.820154    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:33.820276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.820184    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnjgp\" (UniqueName: \"kubernetes.io/projected/17ff5f4b-d5c9-4187-a283-da813d6419fc-kube-api-access-xnjgp\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:33.820276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.820202    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17ff5f4b-d5c9-4187-a283-da813d6419fc-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:33.820276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.820216    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17ff5f4b-d5c9-4187-a283-da813d6419fc-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:33.820276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.820229    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17ff5f4b-d5c9-4187-a283-da813d6419fc-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:33.836702 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.836676    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"]
Oct 30 03:11:33.844181 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:33.844160    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-secondary-42d50b-predictor-7d99dff557-78s7x"]
Oct 30 03:11:34.625041 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:34.625009    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ff5f4b-d5c9-4187-a283-da813d6419fc" path="/var/lib/kubelet/pods/17ff5f4b-d5c9-4187-a283-da813d6419fc/volumes"
Oct 30 03:11:35.241692 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:35.241655    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Oct 30 03:11:35.245846 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:35.245808    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Oct 30 03:11:37.323674 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:37.323605    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Oct 30 03:11:37.324126 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:37.323747    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:11:38.232933 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.232907    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:11:38.357668 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.357587    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-cabundle-cert\") pod \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") "
Oct 30 03:11:38.357668 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.357650    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-proxy-tls\") pod \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") "
Oct 30 03:11:38.358119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.357747    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-rbac-proxy-sar-config\") pod \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") "
Oct 30 03:11:38.358119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.357782    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kserve-provision-location\") pod \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") "
Oct 30 03:11:38.358119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.357802    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-api-access-vjjs5\" (UniqueName: \"kubernetes.io/projected/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-api-access-vjjs5\") pod \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\" (UID: \"a21b1af3-c1c2-4990-bb51-37b5efc00dfe\") "
Oct 30 03:11:38.358119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.358074    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kserve-provision-location" pod "a21b1af3-c1c2-4990-bb51-37b5efc00dfe" (UID: "a21b1af3-c1c2-4990-bb51-37b5efc00dfe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:38.358119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.358074    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-cabundle-cert" pod "a21b1af3-c1c2-4990-bb51-37b5efc00dfe" (UID: "a21b1af3-c1c2-4990-bb51-37b5efc00dfe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:38.358334 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.358183    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-rbac-proxy-sar-config" pod "a21b1af3-c1c2-4990-bb51-37b5efc00dfe" (UID: "a21b1af3-c1c2-4990-bb51-37b5efc00dfe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:38.359861 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.359838    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-proxy-tls" pod "a21b1af3-c1c2-4990-bb51-37b5efc00dfe" (UID: "a21b1af3-c1c2-4990-bb51-37b5efc00dfe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:11:38.359965 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.359924    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-api-access-vjjs5" pod "a21b1af3-c1c2-4990-bb51-37b5efc00dfe" (UID: "a21b1af3-c1c2-4990-bb51-37b5efc00dfe"). InnerVolumeSpecName "kube-api-access-vjjs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:38.458517 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.458481    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjjs5\" (UniqueName: \"kubernetes.io/projected/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-api-access-vjjs5\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:38.458517 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.458512    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:38.458517 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.458524    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:38.458761 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.458533    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:38.458761 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.458541    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a21b1af3-c1c2-4990-bb51-37b5efc00dfe-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:38.789281 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.789249    4885 generic.go:356] "Generic (PLEG): container finished" podID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerID="75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d" exitCode=0
Oct 30 03:11:38.789426 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.789315    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"
Oct 30 03:11:38.789426 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.789320    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" event={"ID":"a21b1af3-c1c2-4990-bb51-37b5efc00dfe","Type":"ContainerDied","Data":"75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d"}
Oct 30 03:11:38.789501 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.789434    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf" event={"ID":"a21b1af3-c1c2-4990-bb51-37b5efc00dfe","Type":"ContainerDied","Data":"5167674a06b9494b47913ab846f008cd813b06c614622fc0875bb0746a735438"}
Oct 30 03:11:38.789501 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.789459    4885 scope.go:122] "RemoveContainer" containerID="f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626"
Oct 30 03:11:38.797543 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.797527    4885 scope.go:122] "RemoveContainer" containerID="75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d"
Oct 30 03:11:38.804791 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.804775    4885 scope.go:122] "RemoveContainer" containerID="f2d96ab74fe2ef09d257bad8c5a643763326167c5c7914c0fe22e94c07118e57"
Oct 30 03:11:38.814167 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.814141    4885 scope.go:122] "RemoveContainer" containerID="f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626"
Oct 30 03:11:38.814475 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:38.814456    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626\": container with ID starting with f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626 not found: ID does not exist" containerID="f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626"
Oct 30 03:11:38.814529 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.814484    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626"} err="failed to get container status \"f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626\": rpc error: code = NotFound desc = could not find container \"f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626\": container with ID starting with f39c37a5e7f00e09ff380d426db3b159c764263f2cf1ee5bb511f088c344f626 not found: ID does not exist"
Oct 30 03:11:38.814529 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.814500    4885 scope.go:122] "RemoveContainer" containerID="75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d"
Oct 30 03:11:38.814800 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:38.814781    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d\": container with ID starting with 75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d not found: ID does not exist" containerID="75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d"
Oct 30 03:11:38.814864 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.814810    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d"} err="failed to get container status \"75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d\": rpc error: code = NotFound desc = could not find container \"75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d\": container with ID starting with 75065dce8db7daf532dda4b9d949bf804cd2a248b8aad7450ce37a0a9d31af0d not found: ID does not exist"
Oct 30 03:11:38.814864 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.814831    4885 scope.go:122] "RemoveContainer" containerID="f2d96ab74fe2ef09d257bad8c5a643763326167c5c7914c0fe22e94c07118e57"
Oct 30 03:11:38.815060 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:38.815043    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d96ab74fe2ef09d257bad8c5a643763326167c5c7914c0fe22e94c07118e57\": container with ID starting with f2d96ab74fe2ef09d257bad8c5a643763326167c5c7914c0fe22e94c07118e57 not found: ID does not exist" containerID="f2d96ab74fe2ef09d257bad8c5a643763326167c5c7914c0fe22e94c07118e57"
Oct 30 03:11:38.815116 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.815078    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d96ab74fe2ef09d257bad8c5a643763326167c5c7914c0fe22e94c07118e57"} err="failed to get container status \"f2d96ab74fe2ef09d257bad8c5a643763326167c5c7914c0fe22e94c07118e57\": rpc error: code = NotFound desc = could not find container \"f2d96ab74fe2ef09d257bad8c5a643763326167c5c7914c0fe22e94c07118e57\": container with ID starting with f2d96ab74fe2ef09d257bad8c5a643763326167c5c7914c0fe22e94c07118e57 not found: ID does not exist"
Oct 30 03:11:38.846516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.846497    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"]
Oct 30 03:11:38.856779 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:38.856755    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-primary-42d50b-predictor-75fb79f7d6-xlhkf"]
Oct 30 03:11:40.624156 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:40.624126    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" path="/var/lib/kubelet/pods/a21b1af3-c1c2-4990-bb51-37b5efc00dfe/volumes"
Oct 30 03:11:41.253900 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.253870    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"]
Oct 30 03:11:41.254226 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254212    4885 cpu_manager.go:414] "containerMap: removing container" podUID="17ff5f4b-d5c9-4187-a283-da813d6419fc" containerName="storage-initializer"
Oct 30 03:11:41.254226 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254226    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ff5f4b-d5c9-4187-a283-da813d6419fc" containerName="storage-initializer"
Oct 30 03:11:41.254361 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254240    4885 cpu_manager.go:414] "containerMap: removing container" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="storage-initializer"
Oct 30 03:11:41.254361 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254248    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="storage-initializer"
Oct 30 03:11:41.254361 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254269    4885 cpu_manager.go:414] "containerMap: removing container" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container"
Oct 30 03:11:41.254361 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254274    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container"
Oct 30 03:11:41.254361 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254281    4885 cpu_manager.go:414] "containerMap: removing container" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kube-rbac-proxy"
Oct 30 03:11:41.254361 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254285    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kube-rbac-proxy"
Oct 30 03:11:41.254361 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254352    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kserve-container"
Oct 30 03:11:41.254599 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254363    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21b1af3-c1c2-4990-bb51-37b5efc00dfe" containerName="kube-rbac-proxy"
Oct 30 03:11:41.254599 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254373    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ff5f4b-d5c9-4187-a283-da813d6419fc" containerName="storage-initializer"
Oct 30 03:11:41.254599 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254439    4885 cpu_manager.go:414] "containerMap: removing container" podUID="17ff5f4b-d5c9-4187-a283-da813d6419fc" containerName="storage-initializer"
Oct 30 03:11:41.254599 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254445    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ff5f4b-d5c9-4187-a283-da813d6419fc" containerName="storage-initializer"
Oct 30 03:11:41.254599 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.254517    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ff5f4b-d5c9-4187-a283-da813d6419fc" containerName="storage-initializer"
Oct 30 03:11:41.257609 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.257589    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.275915 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.275895    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:11:41.278429 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.278409    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-init-fail-89fc1b-kube-rbac-proxy-sar-config\""
Oct 30 03:11:41.278511 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.278472    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:11:41.288744 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.288724    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"fail-s3-sa-89fc1b-dockercfg-4g44b\""
Oct 30 03:11:41.288859 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.288765    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-init-fail-89fc1b-predictor-serving-cert\""
Oct 30 03:11:41.288859 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.288751    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"fail-s3-secret-89fc1b\""
Oct 30 03:11:41.290402 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.290384    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"]
Oct 30 03:11:41.376585 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.376557    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:11:41.381956 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.381935    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.382036 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.381971    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjxjj\" (UniqueName: \"kubernetes.io/projected/2f2100aa-6303-4e77-be80-10efccd5004a-kube-api-access-wjxjj\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.382036 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.381996    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f2100aa-6303-4e77-be80-10efccd5004a-kserve-provision-location\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.382115 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.382088    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2100aa-6303-4e77-be80-10efccd5004a-proxy-tls\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.382147 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.382130    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-cabundle-cert\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.482961 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.482927    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.483118 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.482973    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjxjj\" (UniqueName: \"kubernetes.io/projected/2f2100aa-6303-4e77-be80-10efccd5004a-kube-api-access-wjxjj\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.483118 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.483009    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f2100aa-6303-4e77-be80-10efccd5004a-kserve-provision-location\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.483118 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.483037    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2100aa-6303-4e77-be80-10efccd5004a-proxy-tls\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.483118 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.483082    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-cabundle-cert\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.483487 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.483462    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f2100aa-6303-4e77-be80-10efccd5004a-kserve-provision-location\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.483723 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.483701    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.483834 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.483760    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-cabundle-cert\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.485726 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.485706    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2100aa-6303-4e77-be80-10efccd5004a-proxy-tls\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.505031 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.504970    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjxjj\" (UniqueName: \"kubernetes.io/projected/2f2100aa-6303-4e77-be80-10efccd5004a-kube-api-access-wjxjj\") pod \"isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") " pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.575479 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.575446    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:41.881315 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:41.881286    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"]
Oct 30 03:11:41.884129 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:11:41.884096    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2100aa_6303_4e77_be80_10efccd5004a.slice/crio-092e9e70feafcb3842f02fee0b606157bd9a0184522d9805ced511a730882adb WatchSource:0}: Error finding container 092e9e70feafcb3842f02fee0b606157bd9a0184522d9805ced511a730882adb: Status 404 returned error can't find the container with id 092e9e70feafcb3842f02fee0b606157bd9a0184522d9805ced511a730882adb
Oct 30 03:11:42.323777 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:42.323742    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Oct 30 03:11:42.328869 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:42.328829    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:11:42.329130 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:42.329105    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:11:42.808092 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:42.808061    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn" event={"ID":"2f2100aa-6303-4e77-be80-10efccd5004a","Type":"ContainerStarted","Data":"bc256455caa59269e8161c17fc9b1156d23586536afe5f437935856834ebcc17"}
Oct 30 03:11:42.808254 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:42.808116    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn" event={"ID":"2f2100aa-6303-4e77-be80-10efccd5004a","Type":"ContainerStarted","Data":"092e9e70feafcb3842f02fee0b606157bd9a0184522d9805ced511a730882adb"}
Oct 30 03:11:44.819223 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:44.819199    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn_2f2100aa-6303-4e77-be80-10efccd5004a/storage-initializer/0.log"
Oct 30 03:11:44.819627 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:44.819238    4885 generic.go:356] "Generic (PLEG): container finished" podID="2f2100aa-6303-4e77-be80-10efccd5004a" containerID="bc256455caa59269e8161c17fc9b1156d23586536afe5f437935856834ebcc17" exitCode=1
Oct 30 03:11:44.819627 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:44.819320    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn" event={"ID":"2f2100aa-6303-4e77-be80-10efccd5004a","Type":"ContainerDied","Data":"bc256455caa59269e8161c17fc9b1156d23586536afe5f437935856834ebcc17"}
Oct 30 03:11:45.825230 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:45.825198    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn_2f2100aa-6303-4e77-be80-10efccd5004a/storage-initializer/0.log"
Oct 30 03:11:45.825627 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:45.825292    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn" event={"ID":"2f2100aa-6303-4e77-be80-10efccd5004a","Type":"ContainerStarted","Data":"e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d"}
Oct 30 03:11:46.215406 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:46.215377    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"]
Oct 30 03:11:46.829247 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:46.829193    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn" podUID="2f2100aa-6303-4e77-be80-10efccd5004a" containerName="storage-initializer" containerID="cri-o://e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d" gracePeriod=30
Oct 30 03:11:47.323637 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:47.323578    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Oct 30 03:11:49.670287 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.670263    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn_2f2100aa-6303-4e77-be80-10efccd5004a/storage-initializer/1.log"
Oct 30 03:11:49.670744 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.670730    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn_2f2100aa-6303-4e77-be80-10efccd5004a/storage-initializer/0.log"
Oct 30 03:11:49.673185 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.673168    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:49.843225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.843154    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn_2f2100aa-6303-4e77-be80-10efccd5004a/storage-initializer/1.log"
Oct 30 03:11:49.843566 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.843544    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn_2f2100aa-6303-4e77-be80-10efccd5004a/storage-initializer/0.log"
Oct 30 03:11:49.843664 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.843578    4885 generic.go:356] "Generic (PLEG): container finished" podID="2f2100aa-6303-4e77-be80-10efccd5004a" containerID="e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d" exitCode=1
Oct 30 03:11:49.843701 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.843656    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn" event={"ID":"2f2100aa-6303-4e77-be80-10efccd5004a","Type":"ContainerDied","Data":"e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d"}
Oct 30 03:11:49.843745 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.843713    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn" event={"ID":"2f2100aa-6303-4e77-be80-10efccd5004a","Type":"ContainerDied","Data":"092e9e70feafcb3842f02fee0b606157bd9a0184522d9805ced511a730882adb"}
Oct 30 03:11:49.843745 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.843661    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"
Oct 30 03:11:49.843745 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.843742    4885 scope.go:122] "RemoveContainer" containerID="e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d"
Oct 30 03:11:49.846888 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.846872    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2100aa-6303-4e77-be80-10efccd5004a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f2100aa-6303-4e77-be80-10efccd5004a-kserve-provision-location\") pod \"2f2100aa-6303-4e77-be80-10efccd5004a\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") "
Oct 30 03:11:49.846943 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.846926    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-cabundle-cert\") pod \"2f2100aa-6303-4e77-be80-10efccd5004a\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") "
Oct 30 03:11:49.846992 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.846953    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f2100aa-6303-4e77-be80-10efccd5004a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2100aa-6303-4e77-be80-10efccd5004a-proxy-tls\") pod \"2f2100aa-6303-4e77-be80-10efccd5004a\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") "
Oct 30 03:11:49.846992 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.846975    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2100aa-6303-4e77-be80-10efccd5004a-kube-api-access-wjxjj\" (UniqueName: \"kubernetes.io/projected/2f2100aa-6303-4e77-be80-10efccd5004a-kube-api-access-wjxjj\") pod \"2f2100aa-6303-4e77-be80-10efccd5004a\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") "
Oct 30 03:11:49.847085 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.847015    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-kube-rbac-proxy-sar-config\") pod \"2f2100aa-6303-4e77-be80-10efccd5004a\" (UID: \"2f2100aa-6303-4e77-be80-10efccd5004a\") "
Oct 30 03:11:49.847192 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.847168    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2100aa-6303-4e77-be80-10efccd5004a-kserve-provision-location" pod "2f2100aa-6303-4e77-be80-10efccd5004a" (UID: "2f2100aa-6303-4e77-be80-10efccd5004a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:49.847313 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.847248    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-cabundle-cert" pod "2f2100aa-6303-4e77-be80-10efccd5004a" (UID: "2f2100aa-6303-4e77-be80-10efccd5004a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:49.847428 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.847406    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-kube-rbac-proxy-sar-config" pod "2f2100aa-6303-4e77-be80-10efccd5004a" (UID: "2f2100aa-6303-4e77-be80-10efccd5004a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:49.849247 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.849226    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2100aa-6303-4e77-be80-10efccd5004a-proxy-tls" pod "2f2100aa-6303-4e77-be80-10efccd5004a" (UID: "2f2100aa-6303-4e77-be80-10efccd5004a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:11:49.849445 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.849423    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2100aa-6303-4e77-be80-10efccd5004a-kube-api-access-wjxjj" pod "2f2100aa-6303-4e77-be80-10efccd5004a" (UID: "2f2100aa-6303-4e77-be80-10efccd5004a"). InnerVolumeSpecName "kube-api-access-wjxjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:49.853032 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.853013    4885 scope.go:122] "RemoveContainer" containerID="bc256455caa59269e8161c17fc9b1156d23586536afe5f437935856834ebcc17"
Oct 30 03:11:49.864119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.864097    4885 scope.go:122] "RemoveContainer" containerID="e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d"
Oct 30 03:11:49.864402 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:49.864382    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d\": container with ID starting with e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d not found: ID does not exist" containerID="e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d"
Oct 30 03:11:49.864469 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.864410    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d"} err="failed to get container status \"e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d\": rpc error: code = NotFound desc = could not find container \"e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d\": container with ID starting with e4279fbb81364f6c6ecb8b481b40e79a9f396ca9cc50848b6945ec246a67a93d not found: ID does not exist"
Oct 30 03:11:49.864469 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.864428    4885 scope.go:122] "RemoveContainer" containerID="bc256455caa59269e8161c17fc9b1156d23586536afe5f437935856834ebcc17"
Oct 30 03:11:49.864690 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:49.864673    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc256455caa59269e8161c17fc9b1156d23586536afe5f437935856834ebcc17\": container with ID starting with bc256455caa59269e8161c17fc9b1156d23586536afe5f437935856834ebcc17 not found: ID does not exist" containerID="bc256455caa59269e8161c17fc9b1156d23586536afe5f437935856834ebcc17"
Oct 30 03:11:49.864742 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.864697    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc256455caa59269e8161c17fc9b1156d23586536afe5f437935856834ebcc17"} err="failed to get container status \"bc256455caa59269e8161c17fc9b1156d23586536afe5f437935856834ebcc17\": rpc error: code = NotFound desc = could not find container \"bc256455caa59269e8161c17fc9b1156d23586536afe5f437935856834ebcc17\": container with ID starting with bc256455caa59269e8161c17fc9b1156d23586536afe5f437935856834ebcc17 not found: ID does not exist"
Oct 30 03:11:49.947801 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.947776    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f2100aa-6303-4e77-be80-10efccd5004a-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:49.947801 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.947801    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:49.947995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.947816    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2100aa-6303-4e77-be80-10efccd5004a-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:49.947995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.947829    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjxjj\" (UniqueName: \"kubernetes.io/projected/2f2100aa-6303-4e77-be80-10efccd5004a-kube-api-access-wjxjj\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:49.947995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:49.947841    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f2100aa-6303-4e77-be80-10efccd5004a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:50.210935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:50.210903    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"]
Oct 30 03:11:50.217266 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:50.217236    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-init-fail-89fc1b-predictor-7565844d6f-62bqn"]
Oct 30 03:11:50.628702 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:50.628656    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2100aa-6303-4e77-be80-10efccd5004a" path="/var/lib/kubelet/pods/2f2100aa-6303-4e77-be80-10efccd5004a/volumes"
Oct 30 03:11:51.996235 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:51.996201    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"]
Oct 30 03:11:51.996825 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:51.996549    4885 cpu_manager.go:414] "containerMap: removing container" podUID="2f2100aa-6303-4e77-be80-10efccd5004a" containerName="storage-initializer"
Oct 30 03:11:51.996825 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:51.996559    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2100aa-6303-4e77-be80-10efccd5004a" containerName="storage-initializer"
Oct 30 03:11:51.996825 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:51.996579    4885 cpu_manager.go:414] "containerMap: removing container" podUID="2f2100aa-6303-4e77-be80-10efccd5004a" containerName="storage-initializer"
Oct 30 03:11:51.996825 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:51.996585    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2100aa-6303-4e77-be80-10efccd5004a" containerName="storage-initializer"
Oct 30 03:11:51.996825 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:51.996668    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2100aa-6303-4e77-be80-10efccd5004a" containerName="storage-initializer"
Oct 30 03:11:51.996825 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:51.996684    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2100aa-6303-4e77-be80-10efccd5004a" containerName="storage-initializer"
Oct 30 03:11:52.001568 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.001545    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.026365 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.026339    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:11:52.026537 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.026391    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:11:52.026537 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.026391    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Oct 30 03:11:52.026537 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.026467    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:11:52.026537 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.026477    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:11:52.026537 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.026494    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:11:52.026537 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.026521    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Oct 30 03:11:52.028341 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.028324    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"]
Oct 30 03:11:52.169101 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.169070    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.169249 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.169126    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gvs\" (UniqueName: \"kubernetes.io/projected/e7366444-5938-4050-8c89-5c9812e9d230-kube-api-access-p5gvs\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.169249 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.169210    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7366444-5938-4050-8c89-5c9812e9d230-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.169249 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.169245    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7366444-5938-4050-8c89-5c9812e9d230-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.169353 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.169274    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.269786 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.269717    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7366444-5938-4050-8c89-5c9812e9d230-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.269786 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.269764    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7366444-5938-4050-8c89-5c9812e9d230-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.269980 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.269786    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.269980 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.269823    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.269980 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.269848    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gvs\" (UniqueName: \"kubernetes.io/projected/e7366444-5938-4050-8c89-5c9812e9d230-kube-api-access-p5gvs\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.270170 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.270150    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7366444-5938-4050-8c89-5c9812e9d230-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.270467 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.270448    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.270549 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.270506    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.272446 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.272425    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7366444-5938-4050-8c89-5c9812e9d230-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.291274 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.291249    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gvs\" (UniqueName: \"kubernetes.io/projected/e7366444-5938-4050-8c89-5c9812e9d230-kube-api-access-p5gvs\") pod \"isvc-predictive-sklearn-predictor-57c466c4cf-bkc78\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.318872 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.318850    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:11:52.323096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.323075    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Oct 30 03:11:52.328592 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.328555    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:5000: connect: connection refused"
Oct 30 03:11:52.328716 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.328701    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:11:52.328951 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.328930    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:11:52.329040 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.329027    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:11:52.624952 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.624883    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"]
Oct 30 03:11:52.628126 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:11:52.628090    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7366444_5938_4050_8c89_5c9812e9d230.slice/crio-140212f774c3365b8e41d0d19f58e4bb104c38a358f9fe24b1198c83777a4118 WatchSource:0}: Error finding container 140212f774c3365b8e41d0d19f58e4bb104c38a358f9fe24b1198c83777a4118: Status 404 returned error can't find the container with id 140212f774c3365b8e41d0d19f58e4bb104c38a358f9fe24b1198c83777a4118
Oct 30 03:11:52.859560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.859520    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" event={"ID":"e7366444-5938-4050-8c89-5c9812e9d230","Type":"ContainerStarted","Data":"7bbba17dcad8f71a8fada9f4e5c9aacd980be4da948dbc917542f17103835327"}
Oct 30 03:11:52.859778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:52.859580    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" event={"ID":"e7366444-5938-4050-8c89-5c9812e9d230","Type":"ContainerStarted","Data":"140212f774c3365b8e41d0d19f58e4bb104c38a358f9fe24b1198c83777a4118"}
Oct 30 03:11:55.568766 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.568743    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:11:55.697353 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.697280    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-cabundle-cert\") pod \"9d6783b1-7e11-4004-8c72-742837cb58b7\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") "
Oct 30 03:11:55.697353 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.697332    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-rbac-proxy-sar-config\") pod \"9d6783b1-7e11-4004-8c72-742837cb58b7\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") "
Oct 30 03:11:55.697548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.697374    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d6783b1-7e11-4004-8c72-742837cb58b7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d6783b1-7e11-4004-8c72-742837cb58b7-proxy-tls\") pod \"9d6783b1-7e11-4004-8c72-742837cb58b7\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") "
Oct 30 03:11:55.697548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.697411    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6783b1-7e11-4004-8c72-742837cb58b7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d6783b1-7e11-4004-8c72-742837cb58b7-kserve-provision-location\") pod \"9d6783b1-7e11-4004-8c72-742837cb58b7\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") "
Oct 30 03:11:55.697548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.697428    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-api-access-2jdtv\" (UniqueName: \"kubernetes.io/projected/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-api-access-2jdtv\") pod \"9d6783b1-7e11-4004-8c72-742837cb58b7\" (UID: \"9d6783b1-7e11-4004-8c72-742837cb58b7\") "
Oct 30 03:11:55.697762 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.697738    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-rbac-proxy-sar-config" pod "9d6783b1-7e11-4004-8c72-742837cb58b7" (UID: "9d6783b1-7e11-4004-8c72-742837cb58b7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:55.697823 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.697767    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-cabundle-cert" pod "9d6783b1-7e11-4004-8c72-742837cb58b7" (UID: "9d6783b1-7e11-4004-8c72-742837cb58b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:55.697823 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.697781    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6783b1-7e11-4004-8c72-742837cb58b7-kserve-provision-location" pod "9d6783b1-7e11-4004-8c72-742837cb58b7" (UID: "9d6783b1-7e11-4004-8c72-742837cb58b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:55.699658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.699636    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-api-access-2jdtv" pod "9d6783b1-7e11-4004-8c72-742837cb58b7" (UID: "9d6783b1-7e11-4004-8c72-742837cb58b7"). InnerVolumeSpecName "kube-api-access-2jdtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:55.699658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.699640    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6783b1-7e11-4004-8c72-742837cb58b7-proxy-tls" pod "9d6783b1-7e11-4004-8c72-742837cb58b7" (UID: "9d6783b1-7e11-4004-8c72-742837cb58b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:11:55.798840 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.798821    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:55.798840 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.798840    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:55.798977 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.798854    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d6783b1-7e11-4004-8c72-742837cb58b7-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:55.798977 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.798863    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d6783b1-7e11-4004-8c72-742837cb58b7-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:55.798977 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.798873    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jdtv\" (UniqueName: \"kubernetes.io/projected/9d6783b1-7e11-4004-8c72-742837cb58b7-kube-api-access-2jdtv\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:11:55.875245 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.875215    4885 generic.go:356] "Generic (PLEG): container finished" podID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerID="b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998" exitCode=0
Oct 30 03:11:55.875367 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.875281    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"
Oct 30 03:11:55.875367 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.875300    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" event={"ID":"9d6783b1-7e11-4004-8c72-742837cb58b7","Type":"ContainerDied","Data":"b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998"}
Oct 30 03:11:55.875496 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.875387    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk" event={"ID":"9d6783b1-7e11-4004-8c72-742837cb58b7","Type":"ContainerDied","Data":"8128d0b7022f8093d319ee66b8055ae1d2fb662261ea21983a24d4397b3610c9"}
Oct 30 03:11:55.875496 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.875435    4885 scope.go:122] "RemoveContainer" containerID="b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998"
Oct 30 03:11:55.883954 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.883935    4885 scope.go:122] "RemoveContainer" containerID="d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a"
Oct 30 03:11:55.891837 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.891819    4885 scope.go:122] "RemoveContainer" containerID="5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f"
Oct 30 03:11:55.899442 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.899421    4885 scope.go:122] "RemoveContainer" containerID="4d4a2ea7eaff1c2e3e069ac4a2485d5b9e7283a9f3c0bb3e5908a18cac84605d"
Oct 30 03:11:55.907137 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.907097    4885 scope.go:122] "RemoveContainer" containerID="b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998"
Oct 30 03:11:55.907376 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:55.907355    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998\": container with ID starting with b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998 not found: ID does not exist" containerID="b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998"
Oct 30 03:11:55.907509 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.907382    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998"} err="failed to get container status \"b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998\": rpc error: code = NotFound desc = could not find container \"b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998\": container with ID starting with b30ceb7cf50bd2f9529ab1972da8ddc566bd21e20b18b39b690951f520654998 not found: ID does not exist"
Oct 30 03:11:55.907509 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.907399    4885 scope.go:122] "RemoveContainer" containerID="d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a"
Oct 30 03:11:55.907673 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:55.907652    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a\": container with ID starting with d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a not found: ID does not exist" containerID="d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a"
Oct 30 03:11:55.907722 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.907675    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a"} err="failed to get container status \"d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a\": rpc error: code = NotFound desc = could not find container \"d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a\": container with ID starting with d67aa1ea09081dae44b7169e2041ab66c745f3dc22c63fa710792e78e11ac13a not found: ID does not exist"
Oct 30 03:11:55.907722 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.907694    4885 scope.go:122] "RemoveContainer" containerID="5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f"
Oct 30 03:11:55.907943 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:55.907925    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f\": container with ID starting with 5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f not found: ID does not exist" containerID="5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f"
Oct 30 03:11:55.908017 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.907946    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f"} err="failed to get container status \"5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f\": rpc error: code = NotFound desc = could not find container \"5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f\": container with ID starting with 5177901edb9d5ed939f2079941aa4d7418f3bb4d63e9513fe031387264c8871f not found: ID does not exist"
Oct 30 03:11:55.908017 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.907960    4885 scope.go:122] "RemoveContainer" containerID="4d4a2ea7eaff1c2e3e069ac4a2485d5b9e7283a9f3c0bb3e5908a18cac84605d"
Oct 30 03:11:55.908186 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:55.908168    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d4a2ea7eaff1c2e3e069ac4a2485d5b9e7283a9f3c0bb3e5908a18cac84605d\": container with ID starting with 4d4a2ea7eaff1c2e3e069ac4a2485d5b9e7283a9f3c0bb3e5908a18cac84605d not found: ID does not exist" containerID="4d4a2ea7eaff1c2e3e069ac4a2485d5b9e7283a9f3c0bb3e5908a18cac84605d"
Oct 30 03:11:55.908274 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.908189    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4a2ea7eaff1c2e3e069ac4a2485d5b9e7283a9f3c0bb3e5908a18cac84605d"} err="failed to get container status \"4d4a2ea7eaff1c2e3e069ac4a2485d5b9e7283a9f3c0bb3e5908a18cac84605d\": rpc error: code = NotFound desc = could not find container \"4d4a2ea7eaff1c2e3e069ac4a2485d5b9e7283a9f3c0bb3e5908a18cac84605d\": container with ID starting with 4d4a2ea7eaff1c2e3e069ac4a2485d5b9e7283a9f3c0bb3e5908a18cac84605d not found: ID does not exist"
Oct 30 03:11:55.922058 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.922034    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"]
Oct 30 03:11:55.923001 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:11:55.922977    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:11:55.932357 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:55.932338    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-batcher-custom-predictor-746fb9bb86-hcxwk"]
Oct 30 03:11:56.623669 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:56.623639    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" path="/var/lib/kubelet/pods/9d6783b1-7e11-4004-8c72-742837cb58b7/volumes"
Oct 30 03:11:56.881723 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:56.881641    4885 generic.go:356] "Generic (PLEG): container finished" podID="e7366444-5938-4050-8c89-5c9812e9d230" containerID="7bbba17dcad8f71a8fada9f4e5c9aacd980be4da948dbc917542f17103835327" exitCode=0
Oct 30 03:11:56.881723 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:56.881649    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" event={"ID":"e7366444-5938-4050-8c89-5c9812e9d230","Type":"ContainerDied","Data":"7bbba17dcad8f71a8fada9f4e5c9aacd980be4da948dbc917542f17103835327"}
Oct 30 03:11:56.882951 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:56.882936    4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:11:57.173330 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.173251    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"]
Oct 30 03:11:57.174277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.173755    4885 cpu_manager.go:414] "containerMap: removing container" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent"
Oct 30 03:11:57.174277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.173776    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent"
Oct 30 03:11:57.174277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.173809    4885 cpu_manager.go:414] "containerMap: removing container" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kube-rbac-proxy"
Oct 30 03:11:57.174277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.173816    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kube-rbac-proxy"
Oct 30 03:11:57.174277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.173833    4885 cpu_manager.go:414] "containerMap: removing container" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="storage-initializer"
Oct 30 03:11:57.174277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.173841    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="storage-initializer"
Oct 30 03:11:57.174277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.173863    4885 cpu_manager.go:414] "containerMap: removing container" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container"
Oct 30 03:11:57.174277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.173871    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container"
Oct 30 03:11:57.174277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.173947    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kube-rbac-proxy"
Oct 30 03:11:57.174277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.173962    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="kserve-container"
Oct 30 03:11:57.174277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.173971    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6783b1-7e11-4004-8c72-742837cb58b7" containerName="agent"
Oct 30 03:11:57.177288 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.177268    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"
Oct 30 03:11:57.190691 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.190666    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:11:57.190819 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.190675    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:11:57.190819 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.190727    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:11:57.196097 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.196075    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"]
Oct 30 03:11:57.312422 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.312392    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78c4j\" (UniqueName: \"kubernetes.io/projected/ba2695c8-a9fd-454f-a87b-848f66f613e9-kube-api-access-78c4j\") pod \"proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt\" (UID: \"ba2695c8-a9fd-454f-a87b-848f66f613e9\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"
Oct 30 03:11:57.413467 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.413430    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78c4j\" (UniqueName: \"kubernetes.io/projected/ba2695c8-a9fd-454f-a87b-848f66f613e9-kube-api-access-78c4j\") pod \"proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt\" (UID: \"ba2695c8-a9fd-454f-a87b-848f66f613e9\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"
Oct 30 03:11:57.444420 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.444358    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78c4j\" (UniqueName: \"kubernetes.io/projected/ba2695c8-a9fd-454f-a87b-848f66f613e9-kube-api-access-78c4j\") pod \"proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt\" (UID: \"ba2695c8-a9fd-454f-a87b-848f66f613e9\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"
Oct 30 03:11:57.492161 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.492129    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"
Oct 30 03:11:57.835050 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.834968    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"]
Oct 30 03:11:57.838542 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:11:57.838492    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2695c8_a9fd_454f_a87b_848f66f613e9.slice/crio-976bb5b0c878f01c6b9bc8803fc1ebb86e4c782598515ca84b4ba4273d7b4475 WatchSource:0}: Error finding container 976bb5b0c878f01c6b9bc8803fc1ebb86e4c782598515ca84b4ba4273d7b4475: Status 404 returned error can't find the container with id 976bb5b0c878f01c6b9bc8803fc1ebb86e4c782598515ca84b4ba4273d7b4475
Oct 30 03:11:57.891803 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:11:57.891719    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt" event={"ID":"ba2695c8-a9fd-454f-a87b-848f66f613e9","Type":"ContainerStarted","Data":"976bb5b0c878f01c6b9bc8803fc1ebb86e4c782598515ca84b4ba4273d7b4475"}
Oct 30 03:12:01.923099 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:01.923019    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt" event={"ID":"ba2695c8-a9fd-454f-a87b-848f66f613e9","Type":"ContainerStarted","Data":"b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012"}
Oct 30 03:12:01.924039 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:01.924014    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"
Oct 30 03:12:01.925608 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:01.925580    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt" podUID="ba2695c8-a9fd-454f-a87b-848f66f613e9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/\": dial tcp 10.132.0.42:8080: connect: connection refused"
Oct 30 03:12:01.969160 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:01.969107    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt" podStartSLOduration=1.336306975 podStartE2EDuration="4.969088756s" totalImagesPullingTime="3.632781781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:01.968752025 +0000 UTC m=+1027.478921070" watchObservedRunningTime="2026-10-30 03:12:01.969088756 +0000 UTC m=+1027.479257784"
Oct 30 03:12:02.932022 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:02.931991    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"
Oct 30 03:12:06.951271 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:06.951177    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"]
Oct 30 03:12:06.982151 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:06.982087    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"]
Oct 30 03:12:06.982334 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:06.982213    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:06.995369 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:06.995345    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-ec2e679a-kube-rbac-proxy-sar-config\""
Oct 30 03:12:06.997885 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:06.997845    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9g6\" (UniqueName: \"kubernetes.io/projected/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-api-access-bs9g6\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:06.997995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:06.997895    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2afe1ac-da57-4adf-91e4-759fdc8899ff-proxy-tls\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:06.997995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:06.997977    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-rbac-proxy-sar-config\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:07.005340 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:07.005316    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-ec2e679a-predictor-serving-cert\""
Oct 30 03:12:07.005472 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:07.005456    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-ec2e679a-dockercfg-7wzpp\""
Oct 30 03:12:07.099128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:07.099093    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9g6\" (UniqueName: \"kubernetes.io/projected/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-api-access-bs9g6\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:07.099128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:07.099128    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2afe1ac-da57-4adf-91e4-759fdc8899ff-proxy-tls\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:07.099389 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:07.099192    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-rbac-proxy-sar-config\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:07.099980 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:07.099955    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-rbac-proxy-sar-config\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:07.102278 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:07.102254    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2afe1ac-da57-4adf-91e4-759fdc8899ff-proxy-tls\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:07.121211 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:07.121187    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9g6\" (UniqueName: \"kubernetes.io/projected/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-api-access-bs9g6\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:17.389629 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:17.389572    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:17.415529 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:17.415497    4885 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") device path: \"\"" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:17.490243 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:17.490215    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:17.490428 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:17.490352    4885 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") DevicePath \"\"" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:17.502958 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:17.502931    4885 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") DevicePath \"csi-591bc694eacb18c5f50d63a63cb84faf47fb8016a62f0001c0dae8a9cd9cc5fc\"" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:17.733434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:17.733407    4885 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/53436cc69be3939e87260f4e884991bf9769dd1dee6bd9f8cd75424d9b0e43f0/globalmount\"" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:17.760600 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:17.760572    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:17.802753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:17.802725    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:18.470088 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:18.470056    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"]
Oct 30 03:12:18.481366 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:12:18.481334    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2afe1ac_da57_4adf_91e4_759fdc8899ff.slice/crio-cb0c83a57aa67086b0096812840446750004fbc2431ee8d058f10e02d4d037b0 WatchSource:0}: Error finding container cb0c83a57aa67086b0096812840446750004fbc2431ee8d058f10e02d4d037b0: Status 404 returned error can't find the container with id cb0c83a57aa67086b0096812840446750004fbc2431ee8d058f10e02d4d037b0
Oct 30 03:12:19.013488 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:19.013449    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" event={"ID":"a2afe1ac-da57-4adf-91e4-759fdc8899ff","Type":"ContainerStarted","Data":"09112b89e31112d51cc4051f1c3a078d0291bf10fb02b1258fded4fcf9191fab"}
Oct 30 03:12:19.013753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:19.013522    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:19.013753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:19.013551    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" event={"ID":"a2afe1ac-da57-4adf-91e4-759fdc8899ff","Type":"ContainerStarted","Data":"ed3af4980a0215d598e3c6fe9622e8795fe55497159eec6af13ffe49b45ef406"}
Oct 30 03:12:19.013753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:19.013695    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:19.013753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:19.013744    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" event={"ID":"a2afe1ac-da57-4adf-91e4-759fdc8899ff","Type":"ContainerStarted","Data":"cb0c83a57aa67086b0096812840446750004fbc2431ee8d058f10e02d4d037b0"}
Oct 30 03:12:19.015713 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:19.015693    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:19.103821 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:19.103770    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" podStartSLOduration=16.103754568 podStartE2EDuration="16.103754568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:19.06312031 +0000 UTC m=+1044.573289335" watchObservedRunningTime="2026-10-30 03:12:19.103754568 +0000 UTC m=+1044.613923595"
Oct 30 03:12:20.020876 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:20.020841    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" event={"ID":"e7366444-5938-4050-8c89-5c9812e9d230","Type":"ContainerStarted","Data":"29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08"}
Oct 30 03:12:20.021442 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:20.020952    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" event={"ID":"e7366444-5938-4050-8c89-5c9812e9d230","Type":"ContainerStarted","Data":"93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc"}
Oct 30 03:12:20.021442 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:20.021121    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:12:20.078156 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:20.078096    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podStartSLOduration=2.929511809 podStartE2EDuration="29.078080765s" totalImagesPullingTime="22.285860946s" totalInitContainerRuntime="3.86270801s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:20.076208222 +0000 UTC m=+1045.586377271" watchObservedRunningTime="2026-10-30 03:12:20.078080765 +0000 UTC m=+1045.588249791"
Oct 30 03:12:21.025841 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:21.025804    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:12:21.026960 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:21.026937    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Oct 30 03:12:22.030648 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:22.030594    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Oct 30 03:12:25.921052 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:12:25.921018    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:12:26.030435 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:26.030411    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:27.035417 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:27.035387    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:12:27.035933 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:27.035844    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Oct 30 03:12:34.069112 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.069073    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"]
Oct 30 03:12:34.086333 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.086301    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.096049 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.096012    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"]
Oct 30 03:12:34.128923 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.128891    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a57fc463-c65e-4db8-b805-e423930805cc-proxy-tls\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.129128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.129003    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhnm\" (UniqueName: \"kubernetes.io/projected/a57fc463-c65e-4db8-b805-e423930805cc-kube-api-access-snhnm\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.129128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.129069    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.129373 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.129352    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a57fc463-c65e-4db8-b805-e423930805cc-kube-rbac-proxy-sar-config\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.129458 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.129382    4885 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") DevicePath \"csi-591bc694eacb18c5f50d63a63cb84faf47fb8016a62f0001c0dae8a9cd9cc5fc\"" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.141763 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.141733    4885 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") DevicePath \"csi-591bc694eacb18c5f50d63a63cb84faf47fb8016a62f0001c0dae8a9cd9cc5fc\"" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.162536 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.162509    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.230577 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.230540    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhnm\" (UniqueName: \"kubernetes.io/projected/a57fc463-c65e-4db8-b805-e423930805cc-kube-api-access-snhnm\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.230783 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.230646    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a57fc463-c65e-4db8-b805-e423930805cc-kube-rbac-proxy-sar-config\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.230783 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.230668    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a57fc463-c65e-4db8-b805-e423930805cc-proxy-tls\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.231318 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.231298    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a57fc463-c65e-4db8-b805-e423930805cc-kube-rbac-proxy-sar-config\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.233208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.233184    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a57fc463-c65e-4db8-b805-e423930805cc-proxy-tls\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.252487 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.252462    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhnm\" (UniqueName: \"kubernetes.io/projected/a57fc463-c65e-4db8-b805-e423930805cc-kube-api-access-snhnm\") pod \"proxy-routing-ec2e679a-predictor-f4d68858c-9675x\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") " pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.403501 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.403422    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:34.709456 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:34.709419    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"]
Oct 30 03:12:34.711911 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:12:34.711886    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57fc463_c65e_4db8_b805_e423930805cc.slice/crio-00552081f9a1964517a3929dec8a7cd8c859a8c4941be921023609600515ee60 WatchSource:0}: Error finding container 00552081f9a1964517a3929dec8a7cd8c859a8c4941be921023609600515ee60: Status 404 returned error can't find the container with id 00552081f9a1964517a3929dec8a7cd8c859a8c4941be921023609600515ee60
Oct 30 03:12:35.088451 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:35.088359    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" event={"ID":"a57fc463-c65e-4db8-b805-e423930805cc","Type":"ContainerStarted","Data":"db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a"}
Oct 30 03:12:35.088451 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:35.088423    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" event={"ID":"a57fc463-c65e-4db8-b805-e423930805cc","Type":"ContainerStarted","Data":"1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430"}
Oct 30 03:12:35.088451 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:35.088449    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" event={"ID":"a57fc463-c65e-4db8-b805-e423930805cc","Type":"ContainerStarted","Data":"8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe"}
Oct 30 03:12:35.089038 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:35.088470    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" event={"ID":"a57fc463-c65e-4db8-b805-e423930805cc","Type":"ContainerStarted","Data":"00552081f9a1964517a3929dec8a7cd8c859a8c4941be921023609600515ee60"}
Oct 30 03:12:35.089038 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:35.088500    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:35.129914 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:35.129861    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podStartSLOduration=1.129844081 podStartE2EDuration="1.129844081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:35.128579674 +0000 UTC m=+1060.638748705" watchObservedRunningTime="2026-10-30 03:12:35.129844081 +0000 UTC m=+1060.640013107"
Oct 30 03:12:36.093199 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:36.093159    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:36.093594 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:36.093238    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:36.095637 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:36.095596    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:36.096093 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:36.096076    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:37.036528 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:37.036478    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Oct 30 03:12:43.109722 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:43.109689    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:12:43.194152 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:43.194119    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"]
Oct 30 03:12:43.194440 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:43.194407    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kserve-container" containerID="cri-o://ed3af4980a0215d598e3c6fe9622e8795fe55497159eec6af13ffe49b45ef406" gracePeriod=30
Oct 30 03:12:43.194576 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:43.194476    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kube-rbac-proxy" containerID="cri-o://09112b89e31112d51cc4051f1c3a078d0291bf10fb02b1258fded4fcf9191fab" gracePeriod=30
Oct 30 03:12:44.134956 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:44.134915    4885 generic.go:356] "Generic (PLEG): container finished" podID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerID="09112b89e31112d51cc4051f1c3a078d0291bf10fb02b1258fded4fcf9191fab" exitCode=0
Oct 30 03:12:44.135350 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:44.134977    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" event={"ID":"a2afe1ac-da57-4adf-91e4-759fdc8899ff","Type":"ContainerDied","Data":"09112b89e31112d51cc4051f1c3a078d0291bf10fb02b1258fded4fcf9191fab"}
Oct 30 03:12:46.026118 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:46.026079    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 03:12:47.035847 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:47.035801    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Oct 30 03:12:51.026163 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:51.026120    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 03:12:55.921413 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:12:55.921371    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:12:56.026140 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:56.026106    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 03:12:56.026303 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:56.026247    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:12:57.036304 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:12:57.036262    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Oct 30 03:13:01.026769 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:01.026727    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 03:13:06.026589 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:06.026546    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 03:13:07.036589 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:07.036547    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Oct 30 03:13:11.026772 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:11.026727    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Oct 30 03:13:13.248806 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:13:13.248769    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2afe1ac_da57_4adf_91e4_759fdc8899ff.slice/crio-conmon-ed3af4980a0215d598e3c6fe9622e8795fe55497159eec6af13ffe49b45ef406.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2afe1ac_da57_4adf_91e4_759fdc8899ff.slice/crio-ed3af4980a0215d598e3c6fe9622e8795fe55497159eec6af13ffe49b45ef406.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:13:13.249197 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:13:13.248840    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2afe1ac_da57_4adf_91e4_759fdc8899ff.slice/crio-conmon-ed3af4980a0215d598e3c6fe9622e8795fe55497159eec6af13ffe49b45ef406.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2afe1ac_da57_4adf_91e4_759fdc8899ff.slice/crio-ed3af4980a0215d598e3c6fe9622e8795fe55497159eec6af13ffe49b45ef406.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:13:13.249197 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:13:13.248851    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2afe1ac_da57_4adf_91e4_759fdc8899ff.slice/crio-ed3af4980a0215d598e3c6fe9622e8795fe55497159eec6af13ffe49b45ef406.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2afe1ac_da57_4adf_91e4_759fdc8899ff.slice/crio-conmon-ed3af4980a0215d598e3c6fe9622e8795fe55497159eec6af13ffe49b45ef406.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:13:13.264673 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.264645    4885 generic.go:356] "Generic (PLEG): container finished" podID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerID="ed3af4980a0215d598e3c6fe9622e8795fe55497159eec6af13ffe49b45ef406" exitCode=137
Oct 30 03:13:13.264811 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.264727    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" event={"ID":"a2afe1ac-da57-4adf-91e4-759fdc8899ff","Type":"ContainerDied","Data":"ed3af4980a0215d598e3c6fe9622e8795fe55497159eec6af13ffe49b45ef406"}
Oct 30 03:13:13.382453 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.382432    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:13:13.464569 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.464546    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") "
Oct 30 03:13:13.464729 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.464590    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-api-access-bs9g6\" (UniqueName: \"kubernetes.io/projected/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-api-access-bs9g6\") pod \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") "
Oct 30 03:13:13.464729 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.464670    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2afe1ac-da57-4adf-91e4-759fdc8899ff-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2afe1ac-da57-4adf-91e4-759fdc8899ff-proxy-tls\") pod \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") "
Oct 30 03:13:13.464802 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.464730    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-rbac-proxy-sar-config\") pod \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\" (UID: \"a2afe1ac-da57-4adf-91e4-759fdc8899ff\") "
Oct 30 03:13:13.465162 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.465132    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-rbac-proxy-sar-config" pod "a2afe1ac-da57-4adf-91e4-759fdc8899ff" (UID: "a2afe1ac-da57-4adf-91e4-759fdc8899ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:13.466834 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.466806    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-api-access-bs9g6" pod "a2afe1ac-da57-4adf-91e4-759fdc8899ff" (UID: "a2afe1ac-da57-4adf-91e4-759fdc8899ff"). InnerVolumeSpecName "kube-api-access-bs9g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:13.467208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.467188    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2afe1ac-da57-4adf-91e4-759fdc8899ff-proxy-tls" pod "a2afe1ac-da57-4adf-91e4-759fdc8899ff" (UID: "a2afe1ac-da57-4adf-91e4-759fdc8899ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:13.468201 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.468181    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e" pod "a2afe1ac-da57-4adf-91e4-759fdc8899ff" (UID: "a2afe1ac-da57-4adf-91e4-759fdc8899ff"). InnerVolumeSpecName "pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:13:13.565901 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.565871    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs9g6\" (UniqueName: \"kubernetes.io/projected/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-api-access-bs9g6\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:13:13.565901 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.565895    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2afe1ac-da57-4adf-91e4-759fdc8899ff-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:13:13.565901 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:13.565904    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2afe1ac-da57-4adf-91e4-759fdc8899ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:13:14.271171 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:14.271139    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"
Oct 30 03:13:14.271171 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:14.271152    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9" event={"ID":"a2afe1ac-da57-4adf-91e4-759fdc8899ff","Type":"ContainerDied","Data":"cb0c83a57aa67086b0096812840446750004fbc2431ee8d058f10e02d4d037b0"}
Oct 30 03:13:14.271647 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:14.271244    4885 scope.go:122] "RemoveContainer" containerID="09112b89e31112d51cc4051f1c3a078d0291bf10fb02b1258fded4fcf9191fab"
Oct 30 03:13:14.280318 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:14.280302    4885 scope.go:122] "RemoveContainer" containerID="ed3af4980a0215d598e3c6fe9622e8795fe55497159eec6af13ffe49b45ef406"
Oct 30 03:13:14.328683 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:14.328654    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"]
Oct 30 03:13:14.339872 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:14.339850    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-64d99c5677-4l5k9"]
Oct 30 03:13:14.626247 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:14.626173    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" path="/var/lib/kubelet/pods/a2afe1ac-da57-4adf-91e4-759fdc8899ff/volumes"
Oct 30 03:13:15.673518 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:15.673492    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-4a54dc1685_proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt_ba2695c8-a9fd-454f-a87b-848f66f613e9/kserve-container/0.log"
Oct 30 03:13:15.768147 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:15.768116    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"]
Oct 30 03:13:15.768397 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:15.768365    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt" podUID="ba2695c8-a9fd-454f-a87b-848f66f613e9" containerName="kserve-container" containerID="cri-o://b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012" gracePeriod=30
Oct 30 03:13:16.178460 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:16.178416    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"]
Oct 30 03:13:16.178786 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:16.178748    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="agent" containerID="cri-o://db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a" gracePeriod=30
Oct 30 03:13:16.178964 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:16.178786    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kube-rbac-proxy" containerID="cri-o://1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430" gracePeriod=30
Oct 30 03:13:16.178964 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:16.178735    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kserve-container" containerID="cri-o://8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe" gracePeriod=30
Oct 30 03:13:17.036429 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:17.036391    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Oct 30 03:13:17.287871 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:17.287774    4885 generic.go:356] "Generic (PLEG): container finished" podID="a57fc463-c65e-4db8-b805-e423930805cc" containerID="1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430" exitCode=0
Oct 30 03:13:17.287871 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:17.287845    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" event={"ID":"a57fc463-c65e-4db8-b805-e423930805cc","Type":"ContainerDied","Data":"1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430"}
Oct 30 03:13:18.106047 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:18.106007    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 30 03:13:23.105845 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:23.105802    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 30 03:13:25.922303 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:13:25.922267    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:13:26.094048 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:26.094008    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:13:27.036785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:27.036758    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:13:28.105207 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:28.105168    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 30 03:13:28.105673 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:28.105313    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:13:32.737342 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:32.737310    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"]
Oct 30 03:13:32.737746 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:32.737538    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container" containerID="cri-o://93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc" gracePeriod=30
Oct 30 03:13:32.737746 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:32.737559    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kube-rbac-proxy" containerID="cri-o://29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08" gracePeriod=30
Oct 30 03:13:33.105370 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:33.105276    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 30 03:13:33.373120 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:33.373037    4885 generic.go:356] "Generic (PLEG): container finished" podID="e7366444-5938-4050-8c89-5c9812e9d230" containerID="29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08" exitCode=0
Oct 30 03:13:33.373120 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:33.373101    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" event={"ID":"e7366444-5938-4050-8c89-5c9812e9d230","Type":"ContainerDied","Data":"29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08"}
Oct 30 03:13:36.094015 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:36.093932    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:13:37.031595 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.031553    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Oct 30 03:13:37.035956 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.035929    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Oct 30 03:13:37.181197 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.181170    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:13:37.255576 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.255551    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-kube-rbac-proxy-sar-config\") pod \"e7366444-5938-4050-8c89-5c9812e9d230\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") "
Oct 30 03:13:37.255750 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.255593    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7366444-5938-4050-8c89-5c9812e9d230-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7366444-5938-4050-8c89-5c9812e9d230-kserve-provision-location\") pod \"e7366444-5938-4050-8c89-5c9812e9d230\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") "
Oct 30 03:13:37.255750 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.255661    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-cabundle-cert\") pod \"e7366444-5938-4050-8c89-5c9812e9d230\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") "
Oct 30 03:13:37.255750 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.255725    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7366444-5938-4050-8c89-5c9812e9d230-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7366444-5938-4050-8c89-5c9812e9d230-proxy-tls\") pod \"e7366444-5938-4050-8c89-5c9812e9d230\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") "
Oct 30 03:13:37.255904 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.255771    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7366444-5938-4050-8c89-5c9812e9d230-kube-api-access-p5gvs\" (UniqueName: \"kubernetes.io/projected/e7366444-5938-4050-8c89-5c9812e9d230-kube-api-access-p5gvs\") pod \"e7366444-5938-4050-8c89-5c9812e9d230\" (UID: \"e7366444-5938-4050-8c89-5c9812e9d230\") "
Oct 30 03:13:37.256018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.255986    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7366444-5938-4050-8c89-5c9812e9d230-kserve-provision-location" pod "e7366444-5938-4050-8c89-5c9812e9d230" (UID: "e7366444-5938-4050-8c89-5c9812e9d230"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:37.256155 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.256055    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7366444-5938-4050-8c89-5c9812e9d230-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:13:37.256155 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.256044    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-kube-rbac-proxy-sar-config" pod "e7366444-5938-4050-8c89-5c9812e9d230" (UID: "e7366444-5938-4050-8c89-5c9812e9d230"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:37.256155 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.256089    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-cabundle-cert" pod "e7366444-5938-4050-8c89-5c9812e9d230" (UID: "e7366444-5938-4050-8c89-5c9812e9d230"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:37.257953 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.257925    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7366444-5938-4050-8c89-5c9812e9d230-proxy-tls" pod "e7366444-5938-4050-8c89-5c9812e9d230" (UID: "e7366444-5938-4050-8c89-5c9812e9d230"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:37.258046 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.257988    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7366444-5938-4050-8c89-5c9812e9d230-kube-api-access-p5gvs" pod "e7366444-5938-4050-8c89-5c9812e9d230" (UID: "e7366444-5938-4050-8c89-5c9812e9d230"). InnerVolumeSpecName "kube-api-access-p5gvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:37.356516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.356438    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5gvs\" (UniqueName: \"kubernetes.io/projected/e7366444-5938-4050-8c89-5c9812e9d230-kube-api-access-p5gvs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:13:37.356516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.356469    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:13:37.356516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.356480    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7366444-5938-4050-8c89-5c9812e9d230-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:13:37.356516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.356489    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7366444-5938-4050-8c89-5c9812e9d230-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:13:37.392870 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.392838    4885 generic.go:356] "Generic (PLEG): container finished" podID="e7366444-5938-4050-8c89-5c9812e9d230" containerID="93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc" exitCode=0
Oct 30 03:13:37.393047 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.392903    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"
Oct 30 03:13:37.393047 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.392915    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" event={"ID":"e7366444-5938-4050-8c89-5c9812e9d230","Type":"ContainerDied","Data":"93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc"}
Oct 30 03:13:37.393047 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.392997    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78" event={"ID":"e7366444-5938-4050-8c89-5c9812e9d230","Type":"ContainerDied","Data":"140212f774c3365b8e41d0d19f58e4bb104c38a358f9fe24b1198c83777a4118"}
Oct 30 03:13:37.393047 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.393036    4885 scope.go:122] "RemoveContainer" containerID="29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08"
Oct 30 03:13:37.401876 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.401861    4885 scope.go:122] "RemoveContainer" containerID="93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc"
Oct 30 03:13:37.409287 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.409274    4885 scope.go:122] "RemoveContainer" containerID="7bbba17dcad8f71a8fada9f4e5c9aacd980be4da948dbc917542f17103835327"
Oct 30 03:13:37.416445 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.416421    4885 scope.go:122] "RemoveContainer" containerID="29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08"
Oct 30 03:13:37.416714 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:13:37.416690    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08\": container with ID starting with 29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08 not found: ID does not exist" containerID="29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08"
Oct 30 03:13:37.416765 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.416717    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08"} err="failed to get container status \"29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08\": rpc error: code = NotFound desc = could not find container \"29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08\": container with ID starting with 29db21eff163444622ac71adb0d109b97afe7b94ddb21a089f6dd9aa20228c08 not found: ID does not exist"
Oct 30 03:13:37.416765 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.416733    4885 scope.go:122] "RemoveContainer" containerID="93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc"
Oct 30 03:13:37.416975 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:13:37.416961    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc\": container with ID starting with 93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc not found: ID does not exist" containerID="93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc"
Oct 30 03:13:37.417015 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.416979    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc"} err="failed to get container status \"93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc\": rpc error: code = NotFound desc = could not find container \"93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc\": container with ID starting with 93d22f8008b6f696f816f6551438c53c1eb1e3ba4362d8baa195450e8e096ecc not found: ID does not exist"
Oct 30 03:13:37.417015 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.416991    4885 scope.go:122] "RemoveContainer" containerID="7bbba17dcad8f71a8fada9f4e5c9aacd980be4da948dbc917542f17103835327"
Oct 30 03:13:37.417192 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:13:37.417173    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bbba17dcad8f71a8fada9f4e5c9aacd980be4da948dbc917542f17103835327\": container with ID starting with 7bbba17dcad8f71a8fada9f4e5c9aacd980be4da948dbc917542f17103835327 not found: ID does not exist" containerID="7bbba17dcad8f71a8fada9f4e5c9aacd980be4da948dbc917542f17103835327"
Oct 30 03:13:37.417253 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.417196    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bbba17dcad8f71a8fada9f4e5c9aacd980be4da948dbc917542f17103835327"} err="failed to get container status \"7bbba17dcad8f71a8fada9f4e5c9aacd980be4da948dbc917542f17103835327\": rpc error: code = NotFound desc = could not find container \"7bbba17dcad8f71a8fada9f4e5c9aacd980be4da948dbc917542f17103835327\": container with ID starting with 7bbba17dcad8f71a8fada9f4e5c9aacd980be4da948dbc917542f17103835327 not found: ID does not exist"
Oct 30 03:13:37.439534 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.439513    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"]
Oct 30 03:13:37.443582 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:37.443563    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-sklearn-predictor-57c466c4cf-bkc78"]
Oct 30 03:13:38.105162 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.105130    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 30 03:13:38.368842 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.368766    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"]
Oct 30 03:13:38.369225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369144    4885 cpu_manager.go:414] "containerMap: removing container" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="storage-initializer"
Oct 30 03:13:38.369225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369156    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="storage-initializer"
Oct 30 03:13:38.369225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369169    4885 cpu_manager.go:414] "containerMap: removing container" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container"
Oct 30 03:13:38.369225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369175    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container"
Oct 30 03:13:38.369225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369188    4885 cpu_manager.go:414] "containerMap: removing container" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kube-rbac-proxy"
Oct 30 03:13:38.369225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369194    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kube-rbac-proxy"
Oct 30 03:13:38.369225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369228    4885 cpu_manager.go:414] "containerMap: removing container" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kserve-container"
Oct 30 03:13:38.369548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369234    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kserve-container"
Oct 30 03:13:38.369548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369243    4885 cpu_manager.go:414] "containerMap: removing container" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kube-rbac-proxy"
Oct 30 03:13:38.369548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369248    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kube-rbac-proxy"
Oct 30 03:13:38.369548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369319    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kserve-container"
Oct 30 03:13:38.369548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369329    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7366444-5938-4050-8c89-5c9812e9d230" containerName="kube-rbac-proxy"
Oct 30 03:13:38.369548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369337    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kserve-container"
Oct 30 03:13:38.369548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.369342    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2afe1ac-da57-4adf-91e4-759fdc8899ff" containerName="kube-rbac-proxy"
Oct 30 03:13:38.374032 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.374013    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.397258 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.397238    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:13:38.397496 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.397479    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:13:38.397496 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.397493    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:13:38.397731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.397550    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Oct 30 03:13:38.397731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.397480    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:13:38.397731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.397550    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Oct 30 03:13:38.397731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.397661    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:13:38.399429 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.399410    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"]
Oct 30 03:13:38.466703 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.466672    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.466866 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.466718    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h6hs\" (UniqueName: \"kubernetes.io/projected/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-api-access-5h6hs\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.466866 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.466744    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.466866 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.466806    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.467019 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.466930    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.568082 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.568046    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.568243 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.568115    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.568243 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.568161    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h6hs\" (UniqueName: \"kubernetes.io/projected/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-api-access-5h6hs\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.568243 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.568201    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.568243 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.568227    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.568459 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.568441    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.568789 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.568770    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.568789 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.568781    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.570784 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.570763    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.589841 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.589817    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h6hs\" (UniqueName: \"kubernetes.io/projected/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-api-access-5h6hs\") pod \"isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.624591 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.624534    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7366444-5938-4050-8c89-5c9812e9d230" path="/var/lib/kubelet/pods/e7366444-5938-4050-8c89-5c9812e9d230/volumes"
Oct 30 03:13:38.690356 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.690326    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:38.993571 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:38.993543    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"]
Oct 30 03:13:38.996254 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:13:38.996220    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47532fb4_46e6_4c9c_a7de_62cfd7eb7aa2.slice/crio-e1ff27289faa6a38d59d71214ec2ee1f0ad662c749c65bb65419e348e8674114 WatchSource:0}: Error finding container e1ff27289faa6a38d59d71214ec2ee1f0ad662c749c65bb65419e348e8674114: Status 404 returned error can't find the container with id e1ff27289faa6a38d59d71214ec2ee1f0ad662c749c65bb65419e348e8674114
Oct 30 03:13:39.405339 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:39.405246    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" event={"ID":"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2","Type":"ContainerStarted","Data":"950172629fced7ba5bac624abdb336e0b61196dd9f96c7f1e0333c4cd0c6f632"}
Oct 30 03:13:39.405339 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:39.405308    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" event={"ID":"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2","Type":"ContainerStarted","Data":"e1ff27289faa6a38d59d71214ec2ee1f0ad662c749c65bb65419e348e8674114"}
Oct 30 03:13:43.105351 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:43.105264    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 30 03:13:43.425251 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:43.425172    4885 generic.go:356] "Generic (PLEG): container finished" podID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerID="950172629fced7ba5bac624abdb336e0b61196dd9f96c7f1e0333c4cd0c6f632" exitCode=0
Oct 30 03:13:43.425389 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:43.425245    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" event={"ID":"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2","Type":"ContainerDied","Data":"950172629fced7ba5bac624abdb336e0b61196dd9f96c7f1e0333c4cd0c6f632"}
Oct 30 03:13:44.432016 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:44.431981    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" event={"ID":"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2","Type":"ContainerStarted","Data":"2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8"}
Oct 30 03:13:44.432421 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:44.432048    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" event={"ID":"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2","Type":"ContainerStarted","Data":"e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79"}
Oct 30 03:13:44.432421 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:44.432257    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:44.484657 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:44.484578    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podStartSLOduration=2.745762563 podStartE2EDuration="6.484557486s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.738794923s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:44.482817144 +0000 UTC m=+1129.992986189" watchObservedRunningTime="2026-10-30 03:13:44.484557486 +0000 UTC m=+1129.994726510"
Oct 30 03:13:45.437217 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:45.437182    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:45.438679 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:45.438651    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 30 03:13:45.962639 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:45.962599    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"
Oct 30 03:13:46.093981 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.093898    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:13:46.094143 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.094009    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:13:46.133817 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.133789    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba2695c8-a9fd-454f-a87b-848f66f613e9-kube-api-access-78c4j\" (UniqueName: \"kubernetes.io/projected/ba2695c8-a9fd-454f-a87b-848f66f613e9-kube-api-access-78c4j\") pod \"ba2695c8-a9fd-454f-a87b-848f66f613e9\" (UID: \"ba2695c8-a9fd-454f-a87b-848f66f613e9\") "
Oct 30 03:13:46.137206 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.137176    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2695c8-a9fd-454f-a87b-848f66f613e9-kube-api-access-78c4j" pod "ba2695c8-a9fd-454f-a87b-848f66f613e9" (UID: "ba2695c8-a9fd-454f-a87b-848f66f613e9"). InnerVolumeSpecName "kube-api-access-78c4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:46.234769 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.234738    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78c4j\" (UniqueName: \"kubernetes.io/projected/ba2695c8-a9fd-454f-a87b-848f66f613e9-kube-api-access-78c4j\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:13:46.353394 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.353372    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:13:46.443021 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.442993    4885 generic.go:356] "Generic (PLEG): container finished" podID="ba2695c8-a9fd-454f-a87b-848f66f613e9" containerID="b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012" exitCode=137
Oct 30 03:13:46.443427 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.443037    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"
Oct 30 03:13:46.443427 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.443073    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt" event={"ID":"ba2695c8-a9fd-454f-a87b-848f66f613e9","Type":"ContainerDied","Data":"b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012"}
Oct 30 03:13:46.443427 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.443121    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt" event={"ID":"ba2695c8-a9fd-454f-a87b-848f66f613e9","Type":"ContainerDied","Data":"976bb5b0c878f01c6b9bc8803fc1ebb86e4c782598515ca84b4ba4273d7b4475"}
Oct 30 03:13:46.443427 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.443148    4885 scope.go:122] "RemoveContainer" containerID="b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012"
Oct 30 03:13:46.445213 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.445186    4885 generic.go:356] "Generic (PLEG): container finished" podID="a57fc463-c65e-4db8-b805-e423930805cc" containerID="db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a" exitCode=0
Oct 30 03:13:46.445213 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.445210    4885 generic.go:356] "Generic (PLEG): container finished" podID="a57fc463-c65e-4db8-b805-e423930805cc" containerID="8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe" exitCode=137
Oct 30 03:13:46.445406 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.445246    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"
Oct 30 03:13:46.445406 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.445271    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" event={"ID":"a57fc463-c65e-4db8-b805-e423930805cc","Type":"ContainerDied","Data":"db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a"}
Oct 30 03:13:46.445406 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.445324    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" event={"ID":"a57fc463-c65e-4db8-b805-e423930805cc","Type":"ContainerDied","Data":"8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe"}
Oct 30 03:13:46.445406 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.445347    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x" event={"ID":"a57fc463-c65e-4db8-b805-e423930805cc","Type":"ContainerDied","Data":"00552081f9a1964517a3929dec8a7cd8c859a8c4941be921023609600515ee60"}
Oct 30 03:13:46.445775 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.445754    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 30 03:13:46.451913 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.451783    4885 scope.go:122] "RemoveContainer" containerID="b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012"
Oct 30 03:13:46.452072 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:13:46.452055    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012\": container with ID starting with b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012 not found: ID does not exist" containerID="b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012"
Oct 30 03:13:46.452151 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.452078    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012"} err="failed to get container status \"b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012\": rpc error: code = NotFound desc = could not find container \"b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012\": container with ID starting with b2872ad73dd01f318c03ee0e4ed905a4762742673c5ef139b42c052ebfb7e012 not found: ID does not exist"
Oct 30 03:13:46.452151 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.452093    4885 scope.go:122] "RemoveContainer" containerID="db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a"
Oct 30 03:13:46.460402 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.460380    4885 scope.go:122] "RemoveContainer" containerID="1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430"
Oct 30 03:13:46.468011 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.467994    4885 scope.go:122] "RemoveContainer" containerID="8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe"
Oct 30 03:13:46.474894 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.474873    4885 scope.go:122] "RemoveContainer" containerID="db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a"
Oct 30 03:13:46.475157 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:13:46.475139    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a\": container with ID starting with db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a not found: ID does not exist" containerID="db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a"
Oct 30 03:13:46.475226 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.475167    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a"} err="failed to get container status \"db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a\": rpc error: code = NotFound desc = could not find container \"db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a\": container with ID starting with db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a not found: ID does not exist"
Oct 30 03:13:46.475226 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.475187    4885 scope.go:122] "RemoveContainer" containerID="1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430"
Oct 30 03:13:46.475431 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:13:46.475415    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430\": container with ID starting with 1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430 not found: ID does not exist" containerID="1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430"
Oct 30 03:13:46.475500 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.475440    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430"} err="failed to get container status \"1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430\": rpc error: code = NotFound desc = could not find container \"1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430\": container with ID starting with 1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430 not found: ID does not exist"
Oct 30 03:13:46.475500 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.475458    4885 scope.go:122] "RemoveContainer" containerID="8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe"
Oct 30 03:13:46.475705 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:13:46.475688    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe\": container with ID starting with 8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe not found: ID does not exist" containerID="8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe"
Oct 30 03:13:46.475752 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.475712    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe"} err="failed to get container status \"8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe\": rpc error: code = NotFound desc = could not find container \"8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe\": container with ID starting with 8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe not found: ID does not exist"
Oct 30 03:13:46.475752 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.475731    4885 scope.go:122] "RemoveContainer" containerID="db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a"
Oct 30 03:13:46.476005 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.475986    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a"} err="failed to get container status \"db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a\": rpc error: code = NotFound desc = could not find container \"db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a\": container with ID starting with db00334057dc46e7870b597416f3bf65c47aaaffcb82dbe5f7652bbd6593048a not found: ID does not exist"
Oct 30 03:13:46.476065 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.476006    4885 scope.go:122] "RemoveContainer" containerID="1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430"
Oct 30 03:13:46.476234 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.476217    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430"} err="failed to get container status \"1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430\": rpc error: code = NotFound desc = could not find container \"1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430\": container with ID starting with 1de3ad9e4d6273237b2375bbd233561840ebdff4eb0e9cfdc9344593620be430 not found: ID does not exist"
Oct 30 03:13:46.476283 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.476234    4885 scope.go:122] "RemoveContainer" containerID="8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe"
Oct 30 03:13:46.476447 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.476432    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe"} err="failed to get container status \"8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe\": rpc error: code = NotFound desc = could not find container \"8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe\": container with ID starting with 8e05c10ede3ff2d578a16941461322bc4c81f5a17de39bb6f7a9f1b92e1f8efe not found: ID does not exist"
Oct 30 03:13:46.500959 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.500941    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"]
Oct 30 03:13:46.502904 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.502887    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-sink-b5cd7d5fc-pwjgt"]
Oct 30 03:13:46.536789 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.536764    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a57fc463-c65e-4db8-b805-e423930805cc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a57fc463-c65e-4db8-b805-e423930805cc-kube-rbac-proxy-sar-config\") pod \"a57fc463-c65e-4db8-b805-e423930805cc\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") "
Oct 30 03:13:46.536914 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.536899    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") pod \"a57fc463-c65e-4db8-b805-e423930805cc\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") "
Oct 30 03:13:46.536974 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.536961    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a57fc463-c65e-4db8-b805-e423930805cc-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a57fc463-c65e-4db8-b805-e423930805cc-proxy-tls\") pod \"a57fc463-c65e-4db8-b805-e423930805cc\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") "
Oct 30 03:13:46.537036 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.537017    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57fc463-c65e-4db8-b805-e423930805cc-kube-api-access-snhnm\" (UniqueName: \"kubernetes.io/projected/a57fc463-c65e-4db8-b805-e423930805cc-kube-api-access-snhnm\") pod \"a57fc463-c65e-4db8-b805-e423930805cc\" (UID: \"a57fc463-c65e-4db8-b805-e423930805cc\") "
Oct 30 03:13:46.537113 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.537089    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57fc463-c65e-4db8-b805-e423930805cc-kube-rbac-proxy-sar-config" pod "a57fc463-c65e-4db8-b805-e423930805cc" (UID: "a57fc463-c65e-4db8-b805-e423930805cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:46.537294 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.537270    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a57fc463-c65e-4db8-b805-e423930805cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:13:46.539206 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.539178    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a57fc463-c65e-4db8-b805-e423930805cc-proxy-tls" pod "a57fc463-c65e-4db8-b805-e423930805cc" (UID: "a57fc463-c65e-4db8-b805-e423930805cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:46.539656 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.539628    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57fc463-c65e-4db8-b805-e423930805cc-kube-api-access-snhnm" pod "a57fc463-c65e-4db8-b805-e423930805cc" (UID: "a57fc463-c65e-4db8-b805-e423930805cc"). InnerVolumeSpecName "kube-api-access-snhnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:46.539760 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.539743    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e" pod "a57fc463-c65e-4db8-b805-e423930805cc" (UID: "a57fc463-c65e-4db8-b805-e423930805cc"). InnerVolumeSpecName "pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:13:46.623604 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.623540    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba2695c8-a9fd-454f-a87b-848f66f613e9" path="/var/lib/kubelet/pods/ba2695c8-a9fd-454f-a87b-848f66f613e9/volumes"
Oct 30 03:13:46.637700 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.637677    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a57fc463-c65e-4db8-b805-e423930805cc-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:13:46.637700 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.637697    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snhnm\" (UniqueName: \"kubernetes.io/projected/a57fc463-c65e-4db8-b805-e423930805cc-kube-api-access-snhnm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:13:46.637876 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.637727    4885 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") on node \"ip-10-0-140-132.ec2.internal\" "
Oct 30 03:13:46.647873 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.647859    4885 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e") on node "ip-10-0-140-132.ec2.internal"
Oct 30 03:13:46.738363 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.738333    4885 reconciler_common.go:299] "Volume detached for volume \"pvc-90e3c5a2-639d-4dc6-a24d-cc4d6fedd465\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0321577afb33a281e\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"csi-591bc694eacb18c5f50d63a63cb84faf47fb8016a62f0001c0dae8a9cd9cc5fc\""
Oct 30 03:13:46.781666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.781633    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"]
Oct 30 03:13:46.791078 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:46.791056    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-ec2e679a-predictor-f4d68858c-9675x"]
Oct 30 03:13:47.832541 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.832512    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"]
Oct 30 03:13:47.833008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.832906    4885 cpu_manager.go:414] "containerMap: removing container" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kserve-container"
Oct 30 03:13:47.833008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.832926    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kserve-container"
Oct 30 03:13:47.833008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.832944    4885 cpu_manager.go:414] "containerMap: removing container" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kube-rbac-proxy"
Oct 30 03:13:47.833008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.832949    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kube-rbac-proxy"
Oct 30 03:13:47.833008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.832967    4885 cpu_manager.go:414] "containerMap: removing container" podUID="ba2695c8-a9fd-454f-a87b-848f66f613e9" containerName="kserve-container"
Oct 30 03:13:47.833008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.832973    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2695c8-a9fd-454f-a87b-848f66f613e9" containerName="kserve-container"
Oct 30 03:13:47.833008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.832987    4885 cpu_manager.go:414] "containerMap: removing container" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="agent"
Oct 30 03:13:47.833008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.832994    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="agent"
Oct 30 03:13:47.833281 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.833042    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2695c8-a9fd-454f-a87b-848f66f613e9" containerName="kserve-container"
Oct 30 03:13:47.833281 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.833052    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="agent"
Oct 30 03:13:47.833281 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.833059    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kserve-container"
Oct 30 03:13:47.833281 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.833066    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57fc463-c65e-4db8-b805-e423930805cc" containerName="kube-rbac-proxy"
Oct 30 03:13:47.837436 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.837416    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"
Oct 30 03:13:47.851529 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.851503    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:13:47.861549 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.861520    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:13:47.861549 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.861543    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:13:47.863866 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.863845    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"]
Oct 30 03:13:47.947021 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:47.946985    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsn6h\" (UniqueName: \"kubernetes.io/projected/f06859da-75f7-45c7-bac9-5b2eb74ed25b-kube-api-access-tsn6h\") pod \"proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk\" (UID: \"f06859da-75f7-45c7-bac9-5b2eb74ed25b\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"
Oct 30 03:13:48.047935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:48.047905    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsn6h\" (UniqueName: \"kubernetes.io/projected/f06859da-75f7-45c7-bac9-5b2eb74ed25b-kube-api-access-tsn6h\") pod \"proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk\" (UID: \"f06859da-75f7-45c7-bac9-5b2eb74ed25b\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"
Oct 30 03:13:48.069543 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:48.069514    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsn6h\" (UniqueName: \"kubernetes.io/projected/f06859da-75f7-45c7-bac9-5b2eb74ed25b-kube-api-access-tsn6h\") pod \"proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk\" (UID: \"f06859da-75f7-45c7-bac9-5b2eb74ed25b\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"
Oct 30 03:13:48.150654 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:48.150569    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"
Oct 30 03:13:48.443514 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:48.443481    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"]
Oct 30 03:13:48.446150 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:13:48.446110    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06859da_75f7_45c7_bac9_5b2eb74ed25b.slice/crio-7cae77382ed3ad51e4937269b6e18ad14aa26666811913f905b0c46048681b63 WatchSource:0}: Error finding container 7cae77382ed3ad51e4937269b6e18ad14aa26666811913f905b0c46048681b63: Status 404 returned error can't find the container with id 7cae77382ed3ad51e4937269b6e18ad14aa26666811913f905b0c46048681b63
Oct 30 03:13:48.457665 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:48.457642    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk" event={"ID":"f06859da-75f7-45c7-bac9-5b2eb74ed25b","Type":"ContainerStarted","Data":"7cae77382ed3ad51e4937269b6e18ad14aa26666811913f905b0c46048681b63"}
Oct 30 03:13:48.627656 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:48.626874    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a57fc463-c65e-4db8-b805-e423930805cc" path="/var/lib/kubelet/pods/a57fc463-c65e-4db8-b805-e423930805cc/volumes"
Oct 30 03:13:49.464032 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:49.463988    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk" event={"ID":"f06859da-75f7-45c7-bac9-5b2eb74ed25b","Type":"ContainerStarted","Data":"e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8"}
Oct 30 03:13:49.464420 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:49.464124    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"
Oct 30 03:13:49.466529 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:49.466509    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"
Oct 30 03:13:49.501253 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:49.501203    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk" podStartSLOduration=2.501190363 podStartE2EDuration="2.501190363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:49.499701891 +0000 UTC m=+1135.009870941" watchObservedRunningTime="2026-10-30 03:13:49.501190363 +0000 UTC m=+1135.011359415"
Oct 30 03:13:51.449881 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:51.449848    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:13:51.450396 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:51.450371    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 30 03:13:53.547231 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.547200    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"]
Oct 30 03:13:53.551844 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.551824    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:53.564720 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.564703    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-0d4d0b9a-predictor-serving-cert\""
Oct 30 03:13:53.574760 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.574740    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-0d4d0b9a-kube-rbac-proxy-sar-config\""
Oct 30 03:13:53.574901 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.574883    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-0d4d0b9a-dockercfg-6v9bv\""
Oct 30 03:13:53.576331 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.576313    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"]
Oct 30 03:13:53.590876 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.590852    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-rbac-proxy-sar-config\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:53.590966 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.590895    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwdl7\" (UniqueName: \"kubernetes.io/projected/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-api-access-nwdl7\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:53.591013 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.590980    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19313aaa-6650-4d43-8ac5-c07557fe3375-proxy-tls\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:53.691516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.691489    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwdl7\" (UniqueName: \"kubernetes.io/projected/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-api-access-nwdl7\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:53.691662 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.691526    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19313aaa-6650-4d43-8ac5-c07557fe3375-proxy-tls\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:53.691662 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.691605    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-rbac-proxy-sar-config\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:53.692223 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.692206    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-rbac-proxy-sar-config\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:53.694093 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.694076    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19313aaa-6650-4d43-8ac5-c07557fe3375-proxy-tls\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:53.712784 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:53.712760    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwdl7\" (UniqueName: \"kubernetes.io/projected/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-api-access-nwdl7\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:54.753959 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:54.753931    4885 scope.go:122] "RemoveContainer" containerID="2675f6b84caf48ee5f19ea8501380f80d021c14404db636848d731dea123480c"
Oct 30 03:13:54.761965 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:54.761947    4885 scope.go:122] "RemoveContainer" containerID="f44e25d2d5af424460958fb31f75d5e15a6b9f2c91cf34b006f0bf5506058232"
Oct 30 03:13:54.769220 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:54.769203    4885 scope.go:122] "RemoveContainer" containerID="719a3bb4c26fd2ad9941df4f8fe736703d992798b8b972243c73593e0526d81f"
Oct 30 03:13:55.920963 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:13:55.920928    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:13:59.847960 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:59.847919    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:59.872882 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:59.872853    4885 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") device path: \"\"" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:59.948382 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:59.948343    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:59.948553 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:59.948487    4885 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") DevicePath \"\"" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:13:59.961005 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:13:59.960980    4885 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") DevicePath \"csi-6ccb7b1b192df0ac6e539bfddc10a202d6369c3448b85ef58d37a85351ae9eae\"" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:14:00.139549 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:00.139520    4885 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2729107f4e03447600ab12ac43bfefe6fdbd9b0a0bb6141cc49816c97929d553/globalmount\"" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:14:00.162703 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:00.162675    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:14:00.477689 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:00.477668    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-0d4d0b9a-dockercfg-6v9bv\""
Oct 30 03:14:00.485687 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:00.485666    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:14:00.789873 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:00.789653    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"]
Oct 30 03:14:00.792745 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:14:00.792718    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19313aaa_6650_4d43_8ac5_c07557fe3375.slice/crio-e21c23d9ffe6e5a34a1464bd611593d77878ce165bdcee2c0603c986a7b00dbd WatchSource:0}: Error finding container e21c23d9ffe6e5a34a1464bd611593d77878ce165bdcee2c0603c986a7b00dbd: Status 404 returned error can't find the container with id e21c23d9ffe6e5a34a1464bd611593d77878ce165bdcee2c0603c986a7b00dbd
Oct 30 03:14:01.450988 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:01.450951    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 30 03:14:01.516564 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:01.516528    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" event={"ID":"19313aaa-6650-4d43-8ac5-c07557fe3375","Type":"ContainerStarted","Data":"b70996da20a9d63a829996c1ab90519b612d9c6db445c0b73a32a5f2d0b5f5ce"}
Oct 30 03:14:01.516747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:01.516587    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" event={"ID":"19313aaa-6650-4d43-8ac5-c07557fe3375","Type":"ContainerStarted","Data":"f108b0cac03b7d6ab44e4169c33117f8044d58da9111d6c039c793f6511639ed"}
Oct 30 03:14:01.516747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:01.516635    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" event={"ID":"19313aaa-6650-4d43-8ac5-c07557fe3375","Type":"ContainerStarted","Data":"e21c23d9ffe6e5a34a1464bd611593d77878ce165bdcee2c0603c986a7b00dbd"}
Oct 30 03:14:01.516826 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:01.516746    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:14:01.516826 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:01.516790    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:14:01.518983 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:01.518965    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:14:01.561125 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:01.561074    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" podStartSLOduration=11.56105851 podStartE2EDuration="11.56105851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:01.560927914 +0000 UTC m=+1147.071096965" watchObservedRunningTime="2026-10-30 03:14:01.56105851 +0000 UTC m=+1147.071227537"
Oct 30 03:14:08.529895 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:08.529869    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:14:10.631516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.631482    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"]
Oct 30 03:14:10.635251 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.635235    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.653365 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.653343    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"]
Oct 30 03:14:10.736756 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.736727    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22dxj\" (UniqueName: \"kubernetes.io/projected/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-api-access-22dxj\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.736883 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.736768    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.736928 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.736882    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-rbac-proxy-sar-config\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.736928 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.736885    4885 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") DevicePath \"csi-6ccb7b1b192df0ac6e539bfddc10a202d6369c3448b85ef58d37a85351ae9eae\"" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.737006 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.736929    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784ffb75-c377-4bd6-8d86-dedcc94fbe20-proxy-tls\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.751052 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.751026    4885 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") DevicePath \"csi-6ccb7b1b192df0ac6e539bfddc10a202d6369c3448b85ef58d37a85351ae9eae\"" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.773582 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.773556    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.838305 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.838281    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-rbac-proxy-sar-config\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.838413 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.838311    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784ffb75-c377-4bd6-8d86-dedcc94fbe20-proxy-tls\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.838413 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.838329    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22dxj\" (UniqueName: \"kubernetes.io/projected/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-api-access-22dxj\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.839061 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.839010    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-rbac-proxy-sar-config\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.840847 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.840827    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784ffb75-c377-4bd6-8d86-dedcc94fbe20-proxy-tls\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.861558 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.861541    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22dxj\" (UniqueName: \"kubernetes.io/projected/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-api-access-22dxj\") pod \"proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") " pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:10.952860 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:10.952791    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:11.256853 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:11.256808    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"]
Oct 30 03:14:11.259185 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:14:11.259157    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784ffb75_c377_4bd6_8d86_dedcc94fbe20.slice/crio-7733fa9b4def101d2e0cc18654a824e1415f0d2bcaca36e77c92448af0c65af4 WatchSource:0}: Error finding container 7733fa9b4def101d2e0cc18654a824e1415f0d2bcaca36e77c92448af0c65af4: Status 404 returned error can't find the container with id 7733fa9b4def101d2e0cc18654a824e1415f0d2bcaca36e77c92448af0c65af4
Oct 30 03:14:11.450712 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:11.450677    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 30 03:14:11.566085 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:11.566000    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" event={"ID":"784ffb75-c377-4bd6-8d86-dedcc94fbe20","Type":"ContainerStarted","Data":"8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a"}
Oct 30 03:14:11.566085 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:11.566060    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" event={"ID":"784ffb75-c377-4bd6-8d86-dedcc94fbe20","Type":"ContainerStarted","Data":"2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e"}
Oct 30 03:14:11.566318 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:11.566089    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" event={"ID":"784ffb75-c377-4bd6-8d86-dedcc94fbe20","Type":"ContainerStarted","Data":"3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9"}
Oct 30 03:14:11.566318 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:11.566121    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" event={"ID":"784ffb75-c377-4bd6-8d86-dedcc94fbe20","Type":"ContainerStarted","Data":"7733fa9b4def101d2e0cc18654a824e1415f0d2bcaca36e77c92448af0c65af4"}
Oct 30 03:14:11.566318 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:11.566276    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:11.566487 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:11.566336    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:11.567629 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:11.567582    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.48:8080/\": dial tcp 10.132.0.48:8080: connect: connection refused"
Oct 30 03:14:11.610462 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:11.610408    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podStartSLOduration=1.610393754 podStartE2EDuration="1.610393754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:11.609088766 +0000 UTC m=+1157.119257815" watchObservedRunningTime="2026-10-30 03:14:11.610393754 +0000 UTC m=+1157.120562780"
Oct 30 03:14:12.570986 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:12.570939    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:12.572155 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:12.572134    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:12.572568 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:12.572550    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:19.588916 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:19.588887    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:14:19.678553 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:19.678523    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"]
Oct 30 03:14:19.678860 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:19.678829    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kserve-container" containerID="cri-o://f108b0cac03b7d6ab44e4169c33117f8044d58da9111d6c039c793f6511639ed" gracePeriod=30
Oct 30 03:14:19.679115 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:19.678829    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kube-rbac-proxy" containerID="cri-o://b70996da20a9d63a829996c1ab90519b612d9c6db445c0b73a32a5f2d0b5f5ce" gracePeriod=30
Oct 30 03:14:20.611639 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:20.611588    4885 generic.go:356] "Generic (PLEG): container finished" podID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerID="b70996da20a9d63a829996c1ab90519b612d9c6db445c0b73a32a5f2d0b5f5ce" exitCode=0
Oct 30 03:14:20.612036 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:20.611659    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" event={"ID":"19313aaa-6650-4d43-8ac5-c07557fe3375","Type":"ContainerDied","Data":"b70996da20a9d63a829996c1ab90519b612d9c6db445c0b73a32a5f2d0b5f5ce"}
Oct 30 03:14:21.450425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:21.450384    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 30 03:14:23.525950 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:23.525909    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 30 03:14:25.921642 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:14:25.921583    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:14:28.526230 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:28.526191    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 30 03:14:31.451001 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:31.450965    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 30 03:14:33.526831 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:33.526790    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 30 03:14:33.527211 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:33.526926    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:14:38.526242 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:38.526197    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 30 03:14:41.450768 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:41.450730    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 30 03:14:43.526837 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:43.526793    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 30 03:14:48.526279 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:48.526232    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 30 03:14:49.748649 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:49.748590    4885 generic.go:356] "Generic (PLEG): container finished" podID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerID="f108b0cac03b7d6ab44e4169c33117f8044d58da9111d6c039c793f6511639ed" exitCode=137
Oct 30 03:14:49.749216 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:49.748724    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" event={"ID":"19313aaa-6650-4d43-8ac5-c07557fe3375","Type":"ContainerDied","Data":"f108b0cac03b7d6ab44e4169c33117f8044d58da9111d6c039c793f6511639ed"}
Oct 30 03:14:49.868892 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:49.868866    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:14:49.994243 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:49.994210    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19313aaa-6650-4d43-8ac5-c07557fe3375-proxy-tls\" (UniqueName: \"kubernetes.io/secret/19313aaa-6650-4d43-8ac5-c07557fe3375-proxy-tls\") pod \"19313aaa-6650-4d43-8ac5-c07557fe3375\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") "
Oct 30 03:14:49.994430 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:49.994264    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-rbac-proxy-sar-config\") pod \"19313aaa-6650-4d43-8ac5-c07557fe3375\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") "
Oct 30 03:14:49.994430 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:49.994350    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-api-access-nwdl7\" (UniqueName: \"kubernetes.io/projected/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-api-access-nwdl7\") pod \"19313aaa-6650-4d43-8ac5-c07557fe3375\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") "
Oct 30 03:14:49.994553 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:49.994468    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"19313aaa-6650-4d43-8ac5-c07557fe3375\" (UID: \"19313aaa-6650-4d43-8ac5-c07557fe3375\") "
Oct 30 03:14:49.994706 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:49.994673    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-rbac-proxy-sar-config" pod "19313aaa-6650-4d43-8ac5-c07557fe3375" (UID: "19313aaa-6650-4d43-8ac5-c07557fe3375"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:14:49.994800 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:49.994787    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:14:49.996663 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:49.996635    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19313aaa-6650-4d43-8ac5-c07557fe3375-proxy-tls" pod "19313aaa-6650-4d43-8ac5-c07557fe3375" (UID: "19313aaa-6650-4d43-8ac5-c07557fe3375"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:14:49.996663 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:49.996642    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-api-access-nwdl7" pod "19313aaa-6650-4d43-8ac5-c07557fe3375" (UID: "19313aaa-6650-4d43-8ac5-c07557fe3375"). InnerVolumeSpecName "kube-api-access-nwdl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:14:49.997371 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:49.997354    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2" pod "19313aaa-6650-4d43-8ac5-c07557fe3375" (UID: "19313aaa-6650-4d43-8ac5-c07557fe3375"). InnerVolumeSpecName "pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:14:50.096154 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:50.096117    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19313aaa-6650-4d43-8ac5-c07557fe3375-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:14:50.096154 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:50.096149    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwdl7\" (UniqueName: \"kubernetes.io/projected/19313aaa-6650-4d43-8ac5-c07557fe3375-kube-api-access-nwdl7\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:14:50.754956 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:50.754919    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb" event={"ID":"19313aaa-6650-4d43-8ac5-c07557fe3375","Type":"ContainerDied","Data":"e21c23d9ffe6e5a34a1464bd611593d77878ce165bdcee2c0603c986a7b00dbd"}
Oct 30 03:14:50.755417 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:50.754992    4885 scope.go:122] "RemoveContainer" containerID="b70996da20a9d63a829996c1ab90519b612d9c6db445c0b73a32a5f2d0b5f5ce"
Oct 30 03:14:50.755417 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:50.754933    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"
Oct 30 03:14:50.763647 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:50.763608    4885 scope.go:122] "RemoveContainer" containerID="f108b0cac03b7d6ab44e4169c33117f8044d58da9111d6c039c793f6511639ed"
Oct 30 03:14:50.822481 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:50.822440    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"]
Oct 30 03:14:50.834259 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:50.834222    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-6bfc9d5cc-xt2pb"]
Oct 30 03:14:51.451855 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:51.451821    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:14:52.625212 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:52.625177    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" path="/var/lib/kubelet/pods/19313aaa-6650-4d43-8ac5-c07557fe3375/volumes"
Oct 30 03:14:54.733982 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:54.733958    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:14:54.736522 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:54.736497    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:14:54.739291 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:54.739272    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:14:54.741712 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:54.741696    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:14:55.922539 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:14:55.922505    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:14:57.172947 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:57.172923    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-4a54dc1685_proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v_784ffb75-c377-4bd6-8d86-dedcc94fbe20/kserve-container/0.log"
Oct 30 03:14:57.272421 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:57.272389    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"]
Oct 30 03:14:57.272645 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:57.272596    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk" podUID="f06859da-75f7-45c7-bac9-5b2eb74ed25b" containerName="kserve-container" containerID="cri-o://e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8" gracePeriod=30
Oct 30 03:14:57.648231 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:57.648202    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"]
Oct 30 03:14:57.648519 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:57.648494    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kserve-container" containerID="cri-o://3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9" gracePeriod=30
Oct 30 03:14:57.648792 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:57.648690    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kube-rbac-proxy" containerID="cri-o://2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e" gracePeriod=30
Oct 30 03:14:57.648792 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:57.648712    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="agent" containerID="cri-o://8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a" gracePeriod=30
Oct 30 03:14:57.790280 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:57.790251    4885 generic.go:356] "Generic (PLEG): container finished" podID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerID="2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e" exitCode=0
Oct 30 03:14:57.790407 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:57.790316    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" event={"ID":"784ffb75-c377-4bd6-8d86-dedcc94fbe20","Type":"ContainerDied","Data":"2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e"}
Oct 30 03:14:59.078120 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:59.078082    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"]
Oct 30 03:14:59.078497 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:59.078330    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container" containerID="cri-o://e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79" gracePeriod=30
Oct 30 03:14:59.078497 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:59.078385    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kube-rbac-proxy" containerID="cri-o://2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8" gracePeriod=30
Oct 30 03:14:59.584559 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:59.584519    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Oct 30 03:14:59.800908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:59.800876    4885 generic.go:356] "Generic (PLEG): container finished" podID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerID="2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8" exitCode=0
Oct 30 03:14:59.801071 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:14:59.800945    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" event={"ID":"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2","Type":"ContainerDied","Data":"2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8"}
Oct 30 03:15:01.446507 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:01.446468    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Oct 30 03:15:01.450790 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:01.450768    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 30 03:15:02.571552 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:02.571504    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:15:03.633127 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.633104    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:15:03.693398 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.693336    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-proxy-tls\" (UniqueName: \"kubernetes.io/secret/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-proxy-tls\") pod \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") "
Oct 30 03:15:03.693398 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.693391    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-api-access-5h6hs\" (UniqueName: \"kubernetes.io/projected/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-api-access-5h6hs\") pod \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") "
Oct 30 03:15:03.693594 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.693422    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kserve-provision-location\") pod \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") "
Oct 30 03:15:03.693594 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.693556    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-rbac-proxy-sar-config\") pod \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") "
Oct 30 03:15:03.693738 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.693594    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-cabundle-cert\") pod \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\" (UID: \"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2\") "
Oct 30 03:15:03.693791 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.693763    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kserve-provision-location" pod "47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" (UID: "47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:03.693896 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.693879    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:03.694137 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.694079    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-cabundle-cert" pod "47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" (UID: "47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:03.694306 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.694208    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-rbac-proxy-sar-config" pod "47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" (UID: "47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:03.695674 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.695652    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-proxy-tls" pod "47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" (UID: "47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:03.695753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.695721    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-api-access-5h6hs" pod "47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" (UID: "47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2"). InnerVolumeSpecName "kube-api-access-5h6hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:03.794913 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.794886    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:03.794913 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.794910    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h6hs\" (UniqueName: \"kubernetes.io/projected/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-api-access-5h6hs\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:03.795065 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.794921    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:03.795065 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.794929    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:03.819452 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.819426    4885 generic.go:356] "Generic (PLEG): container finished" podID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerID="e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79" exitCode=0
Oct 30 03:15:03.819567 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.819485    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"
Oct 30 03:15:03.819567 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.819500    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" event={"ID":"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2","Type":"ContainerDied","Data":"e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79"}
Oct 30 03:15:03.819567 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.819546    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v" event={"ID":"47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2","Type":"ContainerDied","Data":"e1ff27289faa6a38d59d71214ec2ee1f0ad662c749c65bb65419e348e8674114"}
Oct 30 03:15:03.819733 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.819565    4885 scope.go:122] "RemoveContainer" containerID="2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8"
Oct 30 03:15:03.828468 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.828450    4885 scope.go:122] "RemoveContainer" containerID="e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79"
Oct 30 03:15:03.836001 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.835981    4885 scope.go:122] "RemoveContainer" containerID="950172629fced7ba5bac624abdb336e0b61196dd9f96c7f1e0333c4cd0c6f632"
Oct 30 03:15:03.843731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.843717    4885 scope.go:122] "RemoveContainer" containerID="2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8"
Oct 30 03:15:03.844021 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:15:03.844000    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8\": container with ID starting with 2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8 not found: ID does not exist" containerID="2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8"
Oct 30 03:15:03.844102 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.844028    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8"} err="failed to get container status \"2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8\": rpc error: code = NotFound desc = could not find container \"2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8\": container with ID starting with 2db3ea4e5bc9476ac348850769ea108ff033577f25bcc677e3691cce84c63da8 not found: ID does not exist"
Oct 30 03:15:03.844102 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.844053    4885 scope.go:122] "RemoveContainer" containerID="e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79"
Oct 30 03:15:03.844306 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:15:03.844292    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79\": container with ID starting with e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79 not found: ID does not exist" containerID="e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79"
Oct 30 03:15:03.844350 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.844309    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79"} err="failed to get container status \"e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79\": rpc error: code = NotFound desc = could not find container \"e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79\": container with ID starting with e3cdc8acd2f96e5275676b51a161ae69f151d8cc61c5caf6d2f3e9b7fa032f79 not found: ID does not exist"
Oct 30 03:15:03.844350 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.844321    4885 scope.go:122] "RemoveContainer" containerID="950172629fced7ba5bac624abdb336e0b61196dd9f96c7f1e0333c4cd0c6f632"
Oct 30 03:15:03.844539 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:15:03.844522    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950172629fced7ba5bac624abdb336e0b61196dd9f96c7f1e0333c4cd0c6f632\": container with ID starting with 950172629fced7ba5bac624abdb336e0b61196dd9f96c7f1e0333c4cd0c6f632 not found: ID does not exist" containerID="950172629fced7ba5bac624abdb336e0b61196dd9f96c7f1e0333c4cd0c6f632"
Oct 30 03:15:03.844581 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.844541    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950172629fced7ba5bac624abdb336e0b61196dd9f96c7f1e0333c4cd0c6f632"} err="failed to get container status \"950172629fced7ba5bac624abdb336e0b61196dd9f96c7f1e0333c4cd0c6f632\": rpc error: code = NotFound desc = could not find container \"950172629fced7ba5bac624abdb336e0b61196dd9f96c7f1e0333c4cd0c6f632\": container with ID starting with 950172629fced7ba5bac624abdb336e0b61196dd9f96c7f1e0333c4cd0c6f632 not found: ID does not exist"
Oct 30 03:15:03.873415 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.873388    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"]
Oct 30 03:15:03.877286 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:03.877266    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-xgboost-predictor-bbd975f9b-d5h5v"]
Oct 30 03:15:04.584747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:04.584710    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Oct 30 03:15:04.623934 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:04.623907    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" path="/var/lib/kubelet/pods/47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2/volumes"
Oct 30 03:15:06.649827 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.649750    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"]
Oct 30 03:15:06.650276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650143    4885 cpu_manager.go:414] "containerMap: removing container" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kserve-container"
Oct 30 03:15:06.650276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650161    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kserve-container"
Oct 30 03:15:06.650276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650176    4885 cpu_manager.go:414] "containerMap: removing container" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container"
Oct 30 03:15:06.650276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650182    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container"
Oct 30 03:15:06.650276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650195    4885 cpu_manager.go:414] "containerMap: removing container" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kube-rbac-proxy"
Oct 30 03:15:06.650276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650200    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kube-rbac-proxy"
Oct 30 03:15:06.650276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650215    4885 cpu_manager.go:414] "containerMap: removing container" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="storage-initializer"
Oct 30 03:15:06.650276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650220    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="storage-initializer"
Oct 30 03:15:06.650276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650234    4885 cpu_manager.go:414] "containerMap: removing container" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kube-rbac-proxy"
Oct 30 03:15:06.650276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650242    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kube-rbac-proxy"
Oct 30 03:15:06.650564 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650300    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kserve-container"
Oct 30 03:15:06.650564 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650308    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kserve-container"
Oct 30 03:15:06.650564 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650314    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="19313aaa-6650-4d43-8ac5-c07557fe3375" containerName="kube-rbac-proxy"
Oct 30 03:15:06.650564 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.650322    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="47532fb4-46e6-4c9c-a7de-62cfd7eb7aa2" containerName="kube-rbac-proxy"
Oct 30 03:15:06.655085 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.655065    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.669908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.669887    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:15:06.681715 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.681692    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:15:06.682004 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.681987    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Oct 30 03:15:06.682098 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.682009    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:15:06.682098 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.682019    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:15:06.682098 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.682084    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Oct 30 03:15:06.682214 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.682117    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:15:06.683739 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.683717    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"]
Oct 30 03:15:06.717741 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.717716    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/926e39e3-f18b-4728-a88f-51a51774be09-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.717854 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.717767    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.717892 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.717872    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926e39e3-f18b-4728-a88f-51a51774be09-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.717932 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.717903    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.717932 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.717921    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6md6\" (UniqueName: \"kubernetes.io/projected/926e39e3-f18b-4728-a88f-51a51774be09-kube-api-access-v6md6\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.818540 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.818506    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.818689 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.818561    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926e39e3-f18b-4728-a88f-51a51774be09-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.818741 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.818683    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.818741 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.818721    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6md6\" (UniqueName: \"kubernetes.io/projected/926e39e3-f18b-4728-a88f-51a51774be09-kube-api-access-v6md6\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.818815 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.818772    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/926e39e3-f18b-4728-a88f-51a51774be09-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.819158 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.819134    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/926e39e3-f18b-4728-a88f-51a51774be09-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.819426 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.819328    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.819426 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.819342    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.821188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.821166    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926e39e3-f18b-4728-a88f-51a51774be09-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.842590 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.842567    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6md6\" (UniqueName: \"kubernetes.io/projected/926e39e3-f18b-4728-a88f-51a51774be09-kube-api-access-v6md6\") pod \"isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:06.971880 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:06.971852    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:07.277087 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:07.277007    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"]
Oct 30 03:15:07.280735 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:15:07.280702    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926e39e3_f18b_4728_a88f_51a51774be09.slice/crio-d99a363471d7da6761021eb756fcc9d732459d4736c94617e3e3e642cb576d6e WatchSource:0}: Error finding container d99a363471d7da6761021eb756fcc9d732459d4736c94617e3e3e642cb576d6e: Status 404 returned error can't find the container with id d99a363471d7da6761021eb756fcc9d732459d4736c94617e3e3e642cb576d6e
Oct 30 03:15:07.839465 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:07.839429    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" event={"ID":"926e39e3-f18b-4728-a88f-51a51774be09","Type":"ContainerStarted","Data":"46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca"}
Oct 30 03:15:07.839899 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:07.839487    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" event={"ID":"926e39e3-f18b-4728-a88f-51a51774be09","Type":"ContainerStarted","Data":"d99a363471d7da6761021eb756fcc9d732459d4736c94617e3e3e642cb576d6e"}
Oct 30 03:15:09.585684 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:09.585604    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Oct 30 03:15:09.586063 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:09.585755    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:15:11.863161 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:11.863129    4885 generic.go:356] "Generic (PLEG): container finished" podID="926e39e3-f18b-4728-a88f-51a51774be09" containerID="46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca" exitCode=0
Oct 30 03:15:11.863553 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:11.863188    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" event={"ID":"926e39e3-f18b-4728-a88f-51a51774be09","Type":"ContainerDied","Data":"46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca"}
Oct 30 03:15:12.571569 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:12.571531    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:15:12.870191 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:12.870107    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" event={"ID":"926e39e3-f18b-4728-a88f-51a51774be09","Type":"ContainerStarted","Data":"7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8"}
Oct 30 03:15:12.870191 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:12.870176    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" event={"ID":"926e39e3-f18b-4728-a88f-51a51774be09","Type":"ContainerStarted","Data":"07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a"}
Oct 30 03:15:12.870634 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:12.870373    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:12.927825 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:12.927768    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" podStartSLOduration=3.209349852 podStartE2EDuration="6.927754539s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.718404687s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:12.927529947 +0000 UTC m=+1218.437698986" watchObservedRunningTime="2026-10-30 03:15:12.927754539 +0000 UTC m=+1218.437923641"
Oct 30 03:15:13.874740 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:13.874715    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:13.876328 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:13.876296    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" podUID="926e39e3-f18b-4728-a88f-51a51774be09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Oct 30 03:15:14.585645 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:14.585585    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Oct 30 03:15:14.880458 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:14.880421    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" podUID="926e39e3-f18b-4728-a88f-51a51774be09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Oct 30 03:15:19.585030 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:19.584989    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Oct 30 03:15:19.885971 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:19.885881    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:19.887128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:19.887111    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:22.572247 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:22.572207    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:15:22.572676 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:22.572335    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:15:24.585083 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:24.585038    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Oct 30 03:15:25.922401 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:15:25.922366    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:15:26.845552 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:26.845520    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"]
Oct 30 03:15:26.845809 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:26.845783    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" podUID="926e39e3-f18b-4728-a88f-51a51774be09" containerName="kserve-container" containerID="cri-o://07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a" gracePeriod=30
Oct 30 03:15:26.845919 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:26.845812    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" podUID="926e39e3-f18b-4728-a88f-51a51774be09" containerName="kube-rbac-proxy" containerID="cri-o://7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8" gracePeriod=30
Oct 30 03:15:27.463962 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.463940    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"
Oct 30 03:15:27.576657 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.576634    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:27.584739 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.584720    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06859da-75f7-45c7-bac9-5b2eb74ed25b-kube-api-access-tsn6h\" (UniqueName: \"kubernetes.io/projected/f06859da-75f7-45c7-bac9-5b2eb74ed25b-kube-api-access-tsn6h\") pod \"f06859da-75f7-45c7-bac9-5b2eb74ed25b\" (UID: \"f06859da-75f7-45c7-bac9-5b2eb74ed25b\") "
Oct 30 03:15:27.586783 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.586764    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06859da-75f7-45c7-bac9-5b2eb74ed25b-kube-api-access-tsn6h" pod "f06859da-75f7-45c7-bac9-5b2eb74ed25b" (UID: "f06859da-75f7-45c7-bac9-5b2eb74ed25b"). InnerVolumeSpecName "kube-api-access-tsn6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:27.685897 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.685874    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/926e39e3-f18b-4728-a88f-51a51774be09-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/926e39e3-f18b-4728-a88f-51a51774be09-kserve-provision-location\") pod \"926e39e3-f18b-4728-a88f-51a51774be09\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") "
Oct 30 03:15:27.686009 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.685917    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-kube-rbac-proxy-sar-config\") pod \"926e39e3-f18b-4728-a88f-51a51774be09\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") "
Oct 30 03:15:27.686009 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.685966    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926e39e3-f18b-4728-a88f-51a51774be09-kube-api-access-v6md6\" (UniqueName: \"kubernetes.io/projected/926e39e3-f18b-4728-a88f-51a51774be09-kube-api-access-v6md6\") pod \"926e39e3-f18b-4728-a88f-51a51774be09\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") "
Oct 30 03:15:27.686009 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.685990    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-cabundle-cert\") pod \"926e39e3-f18b-4728-a88f-51a51774be09\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") "
Oct 30 03:15:27.686158 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.686018    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/926e39e3-f18b-4728-a88f-51a51774be09-proxy-tls\" (UniqueName: \"kubernetes.io/secret/926e39e3-f18b-4728-a88f-51a51774be09-proxy-tls\") pod \"926e39e3-f18b-4728-a88f-51a51774be09\" (UID: \"926e39e3-f18b-4728-a88f-51a51774be09\") "
Oct 30 03:15:27.686220 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.686171    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926e39e3-f18b-4728-a88f-51a51774be09-kserve-provision-location" pod "926e39e3-f18b-4728-a88f-51a51774be09" (UID: "926e39e3-f18b-4728-a88f-51a51774be09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:27.686338 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.686312    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-kube-rbac-proxy-sar-config" pod "926e39e3-f18b-4728-a88f-51a51774be09" (UID: "926e39e3-f18b-4728-a88f-51a51774be09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:27.686399 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.686337    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-cabundle-cert" pod "926e39e3-f18b-4728-a88f-51a51774be09" (UID: "926e39e3-f18b-4728-a88f-51a51774be09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:27.686399 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.686383    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsn6h\" (UniqueName: \"kubernetes.io/projected/f06859da-75f7-45c7-bac9-5b2eb74ed25b-kube-api-access-tsn6h\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:27.686501 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.686407    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/926e39e3-f18b-4728-a88f-51a51774be09-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:27.688089 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.688066    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926e39e3-f18b-4728-a88f-51a51774be09-kube-api-access-v6md6" pod "926e39e3-f18b-4728-a88f-51a51774be09" (UID: "926e39e3-f18b-4728-a88f-51a51774be09"). InnerVolumeSpecName "kube-api-access-v6md6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:27.688173 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.688119    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926e39e3-f18b-4728-a88f-51a51774be09-proxy-tls" pod "926e39e3-f18b-4728-a88f-51a51774be09" (UID: "926e39e3-f18b-4728-a88f-51a51774be09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:27.786820 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.786793    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:27.786820 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.786819    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6md6\" (UniqueName: \"kubernetes.io/projected/926e39e3-f18b-4728-a88f-51a51774be09-kube-api-access-v6md6\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:27.786995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.786828    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/926e39e3-f18b-4728-a88f-51a51774be09-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:27.786995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.786838    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926e39e3-f18b-4728-a88f-51a51774be09-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:27.818987 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.818932    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:15:27.888076 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.888050    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-rbac-proxy-sar-config\") pod \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") "
Oct 30 03:15:27.888220 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.888144    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-api-access-22dxj\" (UniqueName: \"kubernetes.io/projected/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-api-access-22dxj\") pod \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") "
Oct 30 03:15:27.888220 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.888193    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784ffb75-c377-4bd6-8d86-dedcc94fbe20-proxy-tls\" (UniqueName: \"kubernetes.io/secret/784ffb75-c377-4bd6-8d86-dedcc94fbe20-proxy-tls\") pod \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") "
Oct 30 03:15:27.888335 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.888304    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") pod \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\" (UID: \"784ffb75-c377-4bd6-8d86-dedcc94fbe20\") "
Oct 30 03:15:27.888461 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.888440    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-rbac-proxy-sar-config" pod "784ffb75-c377-4bd6-8d86-dedcc94fbe20" (UID: "784ffb75-c377-4bd6-8d86-dedcc94fbe20"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:27.888537 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.888521    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:27.890462 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.890437    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784ffb75-c377-4bd6-8d86-dedcc94fbe20-proxy-tls" pod "784ffb75-c377-4bd6-8d86-dedcc94fbe20" (UID: "784ffb75-c377-4bd6-8d86-dedcc94fbe20"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:27.890557 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.890481    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-api-access-22dxj" pod "784ffb75-c377-4bd6-8d86-dedcc94fbe20" (UID: "784ffb75-c377-4bd6-8d86-dedcc94fbe20"). InnerVolumeSpecName "kube-api-access-22dxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:27.891274 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.891254    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2" pod "784ffb75-c377-4bd6-8d86-dedcc94fbe20" (UID: "784ffb75-c377-4bd6-8d86-dedcc94fbe20"). InnerVolumeSpecName "pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:15:27.945181 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.945148    4885 generic.go:356] "Generic (PLEG): container finished" podID="f06859da-75f7-45c7-bac9-5b2eb74ed25b" containerID="e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8" exitCode=137
Oct 30 03:15:27.945307 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.945196    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"
Oct 30 03:15:27.945307 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.945228    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk" event={"ID":"f06859da-75f7-45c7-bac9-5b2eb74ed25b","Type":"ContainerDied","Data":"e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8"}
Oct 30 03:15:27.945307 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.945280    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk" event={"ID":"f06859da-75f7-45c7-bac9-5b2eb74ed25b","Type":"ContainerDied","Data":"7cae77382ed3ad51e4937269b6e18ad14aa26666811913f905b0c46048681b63"}
Oct 30 03:15:27.945476 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.945307    4885 scope.go:122] "RemoveContainer" containerID="e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8"
Oct 30 03:15:27.947663 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.947636    4885 generic.go:356] "Generic (PLEG): container finished" podID="926e39e3-f18b-4728-a88f-51a51774be09" containerID="7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8" exitCode=0
Oct 30 03:15:27.947663 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.947664    4885 generic.go:356] "Generic (PLEG): container finished" podID="926e39e3-f18b-4728-a88f-51a51774be09" containerID="07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a" exitCode=0
Oct 30 03:15:27.947827 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.947725    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" event={"ID":"926e39e3-f18b-4728-a88f-51a51774be09","Type":"ContainerDied","Data":"7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8"}
Oct 30 03:15:27.947827 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.947761    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"
Oct 30 03:15:27.947827 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.947804    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" event={"ID":"926e39e3-f18b-4728-a88f-51a51774be09","Type":"ContainerDied","Data":"07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a"}
Oct 30 03:15:27.947975 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.947836    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts" event={"ID":"926e39e3-f18b-4728-a88f-51a51774be09","Type":"ContainerDied","Data":"d99a363471d7da6761021eb756fcc9d732459d4736c94617e3e3e642cb576d6e"}
Oct 30 03:15:27.950740 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.950714    4885 generic.go:356] "Generic (PLEG): container finished" podID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerID="8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a" exitCode=0
Oct 30 03:15:27.950740 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.950739    4885 generic.go:356] "Generic (PLEG): container finished" podID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerID="3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9" exitCode=137
Oct 30 03:15:27.950916 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.950791    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"
Oct 30 03:15:27.950916 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.950805    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" event={"ID":"784ffb75-c377-4bd6-8d86-dedcc94fbe20","Type":"ContainerDied","Data":"8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a"}
Oct 30 03:15:27.950916 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.950866    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" event={"ID":"784ffb75-c377-4bd6-8d86-dedcc94fbe20","Type":"ContainerDied","Data":"3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9"}
Oct 30 03:15:27.950916 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.950906    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v" event={"ID":"784ffb75-c377-4bd6-8d86-dedcc94fbe20","Type":"ContainerDied","Data":"7733fa9b4def101d2e0cc18654a824e1415f0d2bcaca36e77c92448af0c65af4"}
Oct 30 03:15:27.958917 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.958899    4885 scope.go:122] "RemoveContainer" containerID="e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8"
Oct 30 03:15:27.959157 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:15:27.959141    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8\": container with ID starting with e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8 not found: ID does not exist" containerID="e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8"
Oct 30 03:15:27.959210 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.959164    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8"} err="failed to get container status \"e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8\": rpc error: code = NotFound desc = could not find container \"e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8\": container with ID starting with e4ba080496a903a040b0b7233531b254c724927bce5863d8c3bea663749a81b8 not found: ID does not exist"
Oct 30 03:15:27.959210 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.959184    4885 scope.go:122] "RemoveContainer" containerID="7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8"
Oct 30 03:15:27.966866 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.966850    4885 scope.go:122] "RemoveContainer" containerID="07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a"
Oct 30 03:15:27.973783 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.973766    4885 scope.go:122] "RemoveContainer" containerID="46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca"
Oct 30 03:15:27.980823 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.980807    4885 scope.go:122] "RemoveContainer" containerID="7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8"
Oct 30 03:15:27.981088 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:15:27.981068    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8\": container with ID starting with 7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8 not found: ID does not exist" containerID="7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8"
Oct 30 03:15:27.981161 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.981098    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8"} err="failed to get container status \"7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8\": rpc error: code = NotFound desc = could not find container \"7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8\": container with ID starting with 7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8 not found: ID does not exist"
Oct 30 03:15:27.981161 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.981119    4885 scope.go:122] "RemoveContainer" containerID="07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a"
Oct 30 03:15:27.981377 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:15:27.981358    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a\": container with ID starting with 07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a not found: ID does not exist" containerID="07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a"
Oct 30 03:15:27.981416 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.981383    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a"} err="failed to get container status \"07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a\": rpc error: code = NotFound desc = could not find container \"07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a\": container with ID starting with 07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a not found: ID does not exist"
Oct 30 03:15:27.981416 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.981398    4885 scope.go:122] "RemoveContainer" containerID="46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca"
Oct 30 03:15:27.981674 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:15:27.981656    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca\": container with ID starting with 46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca not found: ID does not exist" containerID="46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca"
Oct 30 03:15:27.981731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.981679    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca"} err="failed to get container status \"46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca\": rpc error: code = NotFound desc = could not find container \"46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca\": container with ID starting with 46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca not found: ID does not exist"
Oct 30 03:15:27.981731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.981695    4885 scope.go:122] "RemoveContainer" containerID="7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8"
Oct 30 03:15:27.981966 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.981948    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8"} err="failed to get container status \"7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8\": rpc error: code = NotFound desc = could not find container \"7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8\": container with ID starting with 7016a3a6cd0e2c12de47a20099040cb28b659393b94418b78c9e6c1920397ee8 not found: ID does not exist"
Oct 30 03:15:27.982037 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.981967    4885 scope.go:122] "RemoveContainer" containerID="07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a"
Oct 30 03:15:27.982222 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.982204    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a"} err="failed to get container status \"07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a\": rpc error: code = NotFound desc = could not find container \"07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a\": container with ID starting with 07432dfdebc63344d594dd5147389e46b774b38f614accccf882eb5fe10b6e9a not found: ID does not exist"
Oct 30 03:15:27.982271 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.982223    4885 scope.go:122] "RemoveContainer" containerID="46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca"
Oct 30 03:15:27.982439 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.982418    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca"} err="failed to get container status \"46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca\": rpc error: code = NotFound desc = could not find container \"46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca\": container with ID starting with 46e6470ae954a8e0e5d74048146e5ec9c2dd2eb1928babc4a667adf9416481ca not found: ID does not exist"
Oct 30 03:15:27.982506 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.982440    4885 scope.go:122] "RemoveContainer" containerID="8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a"
Oct 30 03:15:27.989276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.989196    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22dxj\" (UniqueName: \"kubernetes.io/projected/784ffb75-c377-4bd6-8d86-dedcc94fbe20-kube-api-access-22dxj\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:27.989276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.989213    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784ffb75-c377-4bd6-8d86-dedcc94fbe20-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:15:27.989276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.989242    4885 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") on node \"ip-10-0-140-132.ec2.internal\" "
Oct 30 03:15:27.989430 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.989340    4885 scope.go:122] "RemoveContainer" containerID="2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e"
Oct 30 03:15:27.995758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.995678    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"]
Oct 30 03:15:27.997861 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.997841    4885 scope.go:122] "RemoveContainer" containerID="3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9"
Oct 30 03:15:27.999607 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.999569    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-sink-5476b9d569-5szbk"]
Oct 30 03:15:27.999607 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:27.999836    4885 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2") on node "ip-10-0-140-132.ec2.internal"
Oct 30 03:15:28.005293 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.005269    4885 scope.go:122] "RemoveContainer" containerID="8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a"
Oct 30 03:15:28.005552 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:15:28.005528    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a\": container with ID starting with 8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a not found: ID does not exist" containerID="8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a"
Oct 30 03:15:28.005600 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.005551    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a"} err="failed to get container status \"8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a\": rpc error: code = NotFound desc = could not find container \"8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a\": container with ID starting with 8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a not found: ID does not exist"
Oct 30 03:15:28.005600 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.005570    4885 scope.go:122] "RemoveContainer" containerID="2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e"
Oct 30 03:15:28.005892 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:15:28.005872    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e\": container with ID starting with 2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e not found: ID does not exist" containerID="2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e"
Oct 30 03:15:28.005960 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.005900    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e"} err="failed to get container status \"2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e\": rpc error: code = NotFound desc = could not find container \"2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e\": container with ID starting with 2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e not found: ID does not exist"
Oct 30 03:15:28.005960 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.005919    4885 scope.go:122] "RemoveContainer" containerID="3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9"
Oct 30 03:15:28.006161 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:15:28.006144    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9\": container with ID starting with 3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9 not found: ID does not exist" containerID="3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9"
Oct 30 03:15:28.006196 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.006166    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9"} err="failed to get container status \"3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9\": rpc error: code = NotFound desc = could not find container \"3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9\": container with ID starting with 3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9 not found: ID does not exist"
Oct 30 03:15:28.006196 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.006182    4885 scope.go:122] "RemoveContainer" containerID="8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a"
Oct 30 03:15:28.006431 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.006412    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a"} err="failed to get container status \"8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a\": rpc error: code = NotFound desc = could not find container \"8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a\": container with ID starting with 8bc95763e82d43983f81a1b70dae00b4078bf9b78a83b060c778456d412cbc9a not found: ID does not exist"
Oct 30 03:15:28.006476 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.006432    4885 scope.go:122] "RemoveContainer" containerID="2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e"
Oct 30 03:15:28.006725 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.006707    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e"} err="failed to get container status \"2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e\": rpc error: code = NotFound desc = could not find container \"2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e\": container with ID starting with 2d2d58429de30e789db6acdda1362a05379afd4cf320512e3bc556c63575057e not found: ID does not exist"
Oct 30 03:15:28.006782 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.006725    4885 scope.go:122] "RemoveContainer" containerID="3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9"
Oct 30 03:15:28.006949 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.006922    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9"} err="failed to get container status \"3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9\": rpc error: code = NotFound desc = could not find container \"3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9\": container with ID starting with 3b4e06b8f33501a4bc8397fee7f1184cdb2c09a2a6e250873e1081d6484843d9 not found: ID does not exist"
Oct 30 03:15:28.034117 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.034081    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"]
Oct 30 03:15:28.037702 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.037683    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-predictor-6fdf9954fc-hb4ts"]
Oct 30 03:15:28.087156 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.087082    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"]
Oct 30 03:15:28.089584 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.089558    4885 reconciler_common.go:299] "Volume detached for volume \"pvc-fe5feb99-4cea-4fba-8a48-a3400ffc457f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a17bedcfb1c3b0e2\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"csi-6ccb7b1b192df0ac6e539bfddc10a202d6369c3448b85ef58d37a85351ae9eae\""
Oct 30 03:15:28.090913 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.090891    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-0d4d0b9a-predictor-86bbfd49d9-r8l6v"]
Oct 30 03:15:28.623674 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.623645    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" path="/var/lib/kubelet/pods/784ffb75-c377-4bd6-8d86-dedcc94fbe20/volumes"
Oct 30 03:15:28.624160 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.624145    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="926e39e3-f18b-4728-a88f-51a51774be09" path="/var/lib/kubelet/pods/926e39e3-f18b-4728-a88f-51a51774be09/volumes"
Oct 30 03:15:28.624678 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:28.624658    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06859da-75f7-45c7-bac9-5b2eb74ed25b" path="/var/lib/kubelet/pods/f06859da-75f7-45c7-bac9-5b2eb74ed25b/volumes"
Oct 30 03:15:29.232045 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232006    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"]
Oct 30 03:15:29.232314 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232301    4885 cpu_manager.go:414] "containerMap: removing container" podUID="926e39e3-f18b-4728-a88f-51a51774be09" containerName="kube-rbac-proxy"
Oct 30 03:15:29.232314 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232315    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926e39e3-f18b-4728-a88f-51a51774be09" containerName="kube-rbac-proxy"
Oct 30 03:15:29.232434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232325    4885 cpu_manager.go:414] "containerMap: removing container" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kube-rbac-proxy"
Oct 30 03:15:29.232434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232330    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kube-rbac-proxy"
Oct 30 03:15:29.232434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232337    4885 cpu_manager.go:414] "containerMap: removing container" podUID="926e39e3-f18b-4728-a88f-51a51774be09" containerName="storage-initializer"
Oct 30 03:15:29.232434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232343    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926e39e3-f18b-4728-a88f-51a51774be09" containerName="storage-initializer"
Oct 30 03:15:29.232434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232353    4885 cpu_manager.go:414] "containerMap: removing container" podUID="f06859da-75f7-45c7-bac9-5b2eb74ed25b" containerName="kserve-container"
Oct 30 03:15:29.232434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232358    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06859da-75f7-45c7-bac9-5b2eb74ed25b" containerName="kserve-container"
Oct 30 03:15:29.232434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232366    4885 cpu_manager.go:414] "containerMap: removing container" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kserve-container"
Oct 30 03:15:29.232434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232370    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kserve-container"
Oct 30 03:15:29.232434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232375    4885 cpu_manager.go:414] "containerMap: removing container" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="agent"
Oct 30 03:15:29.232434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232379    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="agent"
Oct 30 03:15:29.232434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232390    4885 cpu_manager.go:414] "containerMap: removing container" podUID="926e39e3-f18b-4728-a88f-51a51774be09" containerName="kserve-container"
Oct 30 03:15:29.232434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232395    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926e39e3-f18b-4728-a88f-51a51774be09" containerName="kserve-container"
Oct 30 03:15:29.232807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232454    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kube-rbac-proxy"
Oct 30 03:15:29.232807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232463    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06859da-75f7-45c7-bac9-5b2eb74ed25b" containerName="kserve-container"
Oct 30 03:15:29.232807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232469    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="kserve-container"
Oct 30 03:15:29.232807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232475    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="784ffb75-c377-4bd6-8d86-dedcc94fbe20" containerName="agent"
Oct 30 03:15:29.232807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232481    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="926e39e3-f18b-4728-a88f-51a51774be09" containerName="kserve-container"
Oct 30 03:15:29.232807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.232489    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="926e39e3-f18b-4728-a88f-51a51774be09" containerName="kube-rbac-proxy"
Oct 30 03:15:29.237425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.237396    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"
Oct 30 03:15:29.255648 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.254800    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"]
Oct 30 03:15:29.265116 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.265088    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:15:29.265247 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.265148    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:15:29.265247 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.265222    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:15:29.299749 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.299724    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v54kf\" (UniqueName: \"kubernetes.io/projected/ddf021c4-972a-458b-ace0-143ec43ff698-kube-api-access-v54kf\") pod \"proxy-routing-dd4100c6-sink-689dc4c4c-grrzn\" (UID: \"ddf021c4-972a-458b-ace0-143ec43ff698\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"
Oct 30 03:15:29.400503 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.400472    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v54kf\" (UniqueName: \"kubernetes.io/projected/ddf021c4-972a-458b-ace0-143ec43ff698-kube-api-access-v54kf\") pod \"proxy-routing-dd4100c6-sink-689dc4c4c-grrzn\" (UID: \"ddf021c4-972a-458b-ace0-143ec43ff698\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"
Oct 30 03:15:29.424003 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.423982    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v54kf\" (UniqueName: \"kubernetes.io/projected/ddf021c4-972a-458b-ace0-143ec43ff698-kube-api-access-v54kf\") pod \"proxy-routing-dd4100c6-sink-689dc4c4c-grrzn\" (UID: \"ddf021c4-972a-458b-ace0-143ec43ff698\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"
Oct 30 03:15:29.551835 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.551762    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"
Oct 30 03:15:29.850882 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.850808    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"]
Oct 30 03:15:29.853851 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:15:29.853824    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf021c4_972a_458b_ace0_143ec43ff698.slice/crio-2a0dabc61516d431aa12f2e08202e8e4e0af77bce1b3bec850626688c402cb83 WatchSource:0}: Error finding container 2a0dabc61516d431aa12f2e08202e8e4e0af77bce1b3bec850626688c402cb83: Status 404 returned error can't find the container with id 2a0dabc61516d431aa12f2e08202e8e4e0af77bce1b3bec850626688c402cb83
Oct 30 03:15:29.963599 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.963565    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn" event={"ID":"ddf021c4-972a-458b-ace0-143ec43ff698","Type":"ContainerStarted","Data":"4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee"}
Oct 30 03:15:29.963844 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:29.963675    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn" event={"ID":"ddf021c4-972a-458b-ace0-143ec43ff698","Type":"ContainerStarted","Data":"2a0dabc61516d431aa12f2e08202e8e4e0af77bce1b3bec850626688c402cb83"}
Oct 30 03:15:30.003026 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.002965    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn" podStartSLOduration=1.002946662 podStartE2EDuration="1.002946662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:30.002242345 +0000 UTC m=+1235.512411371" watchObservedRunningTime="2026-10-30 03:15:30.002946662 +0000 UTC m=+1235.513115689"
Oct 30 03:15:30.360715 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.360686    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"]
Oct 30 03:15:30.364446 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.364425    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.381959 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.381938    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Oct 30 03:15:30.382110 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.382095    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Oct 30 03:15:30.385642 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.385626    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:15:30.385742 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.385725    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:15:30.387859 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.387838    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"]
Oct 30 03:15:30.482634 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.482595    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:15:30.482795 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.482595    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:15:30.482795 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.482683    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:15:30.510027 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.509998    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcbk2\" (UniqueName: \"kubernetes.io/projected/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-api-access-wcbk2\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.510175 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.510036    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.510175 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.510062    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55acea28-3670-4eb8-81e4-8cadf898cf84-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.510269 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.510180    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.510269 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.510248    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55acea28-3670-4eb8-81e4-8cadf898cf84-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.611063 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.610984    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcbk2\" (UniqueName: \"kubernetes.io/projected/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-api-access-wcbk2\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.611063 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.611030    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.611063 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.611051    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55acea28-3670-4eb8-81e4-8cadf898cf84-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.611348 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.611087    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.611348 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.611129    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55acea28-3670-4eb8-81e4-8cadf898cf84-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.611578 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.611553    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55acea28-3670-4eb8-81e4-8cadf898cf84-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.611785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.611764    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.611785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.611777    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.613916 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.613895    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55acea28-3670-4eb8-81e4-8cadf898cf84-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.634752 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.634726    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcbk2\" (UniqueName: \"kubernetes.io/projected/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-api-access-wcbk2\") pod \"isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.681706 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.681680    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:30.968516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.968486    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"
Oct 30 03:15:30.970865 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.970843    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"
Oct 30 03:15:30.986929 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:30.986909    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"]
Oct 30 03:15:30.988913 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:15:30.988889    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55acea28_3670_4eb8_81e4_8cadf898cf84.slice/crio-cb58682cc380b73c27ec1f040eb6f707e3538bfc7862a2b09ee4bb998752b25b WatchSource:0}: Error finding container cb58682cc380b73c27ec1f040eb6f707e3538bfc7862a2b09ee4bb998752b25b: Status 404 returned error can't find the container with id cb58682cc380b73c27ec1f040eb6f707e3538bfc7862a2b09ee4bb998752b25b
Oct 30 03:15:31.974405 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:31.974363    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" event={"ID":"55acea28-3670-4eb8-81e4-8cadf898cf84","Type":"ContainerStarted","Data":"2c807b97b044de3932bd18e4a2e1cbdeeb0f27f6f3cca469e00f01098bceac0a"}
Oct 30 03:15:31.974835 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:31.974434    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" event={"ID":"55acea28-3670-4eb8-81e4-8cadf898cf84","Type":"ContainerStarted","Data":"cb58682cc380b73c27ec1f040eb6f707e3538bfc7862a2b09ee4bb998752b25b"}
Oct 30 03:15:34.988491 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:34.988457    4885 generic.go:356] "Generic (PLEG): container finished" podID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerID="2c807b97b044de3932bd18e4a2e1cbdeeb0f27f6f3cca469e00f01098bceac0a" exitCode=0
Oct 30 03:15:34.988875 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:34.988526    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" event={"ID":"55acea28-3670-4eb8-81e4-8cadf898cf84","Type":"ContainerDied","Data":"2c807b97b044de3932bd18e4a2e1cbdeeb0f27f6f3cca469e00f01098bceac0a"}
Oct 30 03:15:35.058116 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.058093    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"]
Oct 30 03:15:35.062628 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.062590    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:35.080007 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.079982    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-dd4100c6-kube-rbac-proxy-sar-config\""
Oct 30 03:15:35.080359 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.080323    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-dd4100c6-predictor-serving-cert\""
Oct 30 03:15:35.080359 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.080324    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-dd4100c6-dockercfg-qcxr7\""
Oct 30 03:15:35.082080 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.082036    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"]
Oct 30 03:15:35.147151 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.147102    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-rbac-proxy-sar-config\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:35.147370 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.147285    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jkzh\" (UniqueName: \"kubernetes.io/projected/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-api-access-2jkzh\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:35.147451 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.147377    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/345b910d-7a8d-4af2-bb0f-3b504c5df63e-proxy-tls\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:35.248456 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.248379    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-rbac-proxy-sar-config\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:35.248456 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.248441    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jkzh\" (UniqueName: \"kubernetes.io/projected/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-api-access-2jkzh\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:35.248702 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.248476    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/345b910d-7a8d-4af2-bb0f-3b504c5df63e-proxy-tls\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:35.249041 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.249017    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-rbac-proxy-sar-config\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:35.250986 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.250967    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/345b910d-7a8d-4af2-bb0f-3b504c5df63e-proxy-tls\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:35.271582 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.271555    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jkzh\" (UniqueName: \"kubernetes.io/projected/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-api-access-2jkzh\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:35.995220 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.995185    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" event={"ID":"55acea28-3670-4eb8-81e4-8cadf898cf84","Type":"ContainerStarted","Data":"1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf"}
Oct 30 03:15:35.995698 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.995256    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" event={"ID":"55acea28-3670-4eb8-81e4-8cadf898cf84","Type":"ContainerStarted","Data":"9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8"}
Oct 30 03:15:35.995698 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:35.995469    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:36.052643 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:36.052573    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" podStartSLOduration=2.514711214 podStartE2EDuration="6.052559163s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.537847949s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:36.051825371 +0000 UTC m=+1241.561994414" watchObservedRunningTime="2026-10-30 03:15:36.052559163 +0000 UTC m=+1241.562728192"
Oct 30 03:15:37.000465 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:37.000432    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:41.805803 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:41.805768    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6d9df4c4-f698-4e8c-b72f-13f2e5364720\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:41.834167 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:41.834142    4885 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6d9df4c4-f698-4e8c-b72f-13f2e5364720\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") device path: \"\"" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:41.906586 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:41.906560    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6d9df4c4-f698-4e8c-b72f-13f2e5364720\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:41.906764 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:41.906700    4885 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6d9df4c4-f698-4e8c-b72f-13f2e5364720\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") DevicePath \"\"" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:41.920802 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:41.920779    4885 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6d9df4c4-f698-4e8c-b72f-13f2e5364720\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") DevicePath \"csi-2563f79b13ff69d913cdeed25c9ccae8535d0f2ec09b5da363f0ebb72246a545\"" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:42.100591 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:42.100563    4885 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6d9df4c4-f698-4e8c-b72f-13f2e5364720\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6381a46f7c85fbd8600dece2f34cbc4fa63e3dcf157471523b10d2a31d14dc64/globalmount\"" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:42.124573 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:42.124550    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6d9df4c4-f698-4e8c-b72f-13f2e5364720\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684\") pod \"proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:42.293089 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:42.293058    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:42.594465 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:42.594431    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"]
Oct 30 03:15:42.597640 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:15:42.597592    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345b910d_7a8d_4af2_bb0f_3b504c5df63e.slice/crio-72992b78c95beba0b86fb9b6e84d9810199c7e183c2efd78498dcec77841750d WatchSource:0}: Error finding container 72992b78c95beba0b86fb9b6e84d9810199c7e183c2efd78498dcec77841750d: Status 404 returned error can't find the container with id 72992b78c95beba0b86fb9b6e84d9810199c7e183c2efd78498dcec77841750d
Oct 30 03:15:43.010215 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:43.010185    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:15:43.027590 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:43.027559    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" event={"ID":"345b910d-7a8d-4af2-bb0f-3b504c5df63e","Type":"ContainerStarted","Data":"b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520"}
Oct 30 03:15:43.027763 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:43.027645    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" event={"ID":"345b910d-7a8d-4af2-bb0f-3b504c5df63e","Type":"ContainerStarted","Data":"3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b"}
Oct 30 03:15:43.027763 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:43.027673    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" event={"ID":"345b910d-7a8d-4af2-bb0f-3b504c5df63e","Type":"ContainerStarted","Data":"72992b78c95beba0b86fb9b6e84d9810199c7e183c2efd78498dcec77841750d"}
Oct 30 03:15:43.027763 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:43.027703    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:43.027763 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:43.027729    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:43.029680 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:43.029659    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:43.111099 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:43.111048    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" podStartSLOduration=11.111033631 podStartE2EDuration="11.111033631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:43.10994743 +0000 UTC m=+1248.620116493" watchObservedRunningTime="2026-10-30 03:15:43.111033631 +0000 UTC m=+1248.621202656"
Oct 30 03:15:50.040111 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:15:50.040081    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:15:55.921587 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:15:55.921549    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:16:13.011598 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:13.011555    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 03:16:22.695677 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:22.695646    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"]
Oct 30 03:16:22.696181 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:22.695866    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn" podUID="ddf021c4-972a-458b-ace0-143ec43ff698" containerName="kserve-container" containerID="cri-o://4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee" gracePeriod=30
Oct 30 03:16:23.011414 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:23.011375    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 03:16:23.147310 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:23.147278    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"]
Oct 30 03:16:23.147559 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:23.147535    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kserve-container" containerID="cri-o://3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b" gracePeriod=30
Oct 30 03:16:23.147705 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:23.147676    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kube-rbac-proxy" containerID="cri-o://b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520" gracePeriod=30
Oct 30 03:16:24.209757 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:24.209721    4885 generic.go:356] "Generic (PLEG): container finished" podID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerID="b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520" exitCode=0
Oct 30 03:16:24.210179 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:24.209797    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" event={"ID":"345b910d-7a8d-4af2-bb0f-3b504c5df63e","Type":"ContainerDied","Data":"b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520"}
Oct 30 03:16:25.036039 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:25.035993    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 03:16:25.922170 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:16:25.922137    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:16:30.036875 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:30.036829    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 03:16:33.011380 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:33.011336    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 03:16:35.035833 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:35.035780    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 03:16:35.036328 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:35.035934    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:16:40.035890 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:40.035797    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 03:16:43.440996 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:43.440962    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:16:45.035764 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:45.035719    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 03:16:50.035888 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:50.035846    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Oct 30 03:16:50.606439 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:50.606406    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"]
Oct 30 03:16:50.606765 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:50.606715    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kserve-container" containerID="cri-o://9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8" gracePeriod=30
Oct 30 03:16:50.606862 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:50.606793    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kube-rbac-proxy" containerID="cri-o://1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf" gracePeriod=30
Oct 30 03:16:51.330777 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:51.330747    4885 generic.go:356] "Generic (PLEG): container finished" podID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerID="1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf" exitCode=0
Oct 30 03:16:51.331146 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:51.330816    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" event={"ID":"55acea28-3670-4eb8-81e4-8cadf898cf84","Type":"ContainerDied","Data":"1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf"}
Oct 30 03:16:52.886042 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:52.886013    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"
Oct 30 03:16:53.002410 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.002384    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf021c4-972a-458b-ace0-143ec43ff698-kube-api-access-v54kf\" (UniqueName: \"kubernetes.io/projected/ddf021c4-972a-458b-ace0-143ec43ff698-kube-api-access-v54kf\") pod \"ddf021c4-972a-458b-ace0-143ec43ff698\" (UID: \"ddf021c4-972a-458b-ace0-143ec43ff698\") "
Oct 30 03:16:53.004554 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.004529    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf021c4-972a-458b-ace0-143ec43ff698-kube-api-access-v54kf" pod "ddf021c4-972a-458b-ace0-143ec43ff698" (UID: "ddf021c4-972a-458b-ace0-143ec43ff698"). InnerVolumeSpecName "kube-api-access-v54kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:53.005848 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.005819    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 30 03:16:53.011240 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.011208    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 30 03:16:53.103477 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.103452    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v54kf\" (UniqueName: \"kubernetes.io/projected/ddf021c4-972a-458b-ace0-143ec43ff698-kube-api-access-v54kf\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:16:53.341935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.341907    4885 generic.go:356] "Generic (PLEG): container finished" podID="ddf021c4-972a-458b-ace0-143ec43ff698" containerID="4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee" exitCode=137
Oct 30 03:16:53.342083 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.341945    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"
Oct 30 03:16:53.342083 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.341990    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn" event={"ID":"ddf021c4-972a-458b-ace0-143ec43ff698","Type":"ContainerDied","Data":"4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee"}
Oct 30 03:16:53.342083 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.342041    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn" event={"ID":"ddf021c4-972a-458b-ace0-143ec43ff698","Type":"ContainerDied","Data":"2a0dabc61516d431aa12f2e08202e8e4e0af77bce1b3bec850626688c402cb83"}
Oct 30 03:16:53.342083 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.342068    4885 scope.go:122] "RemoveContainer" containerID="4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee"
Oct 30 03:16:53.342338 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.342261    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:16:53.343707 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.343683    4885 generic.go:356] "Generic (PLEG): container finished" podID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerID="3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b" exitCode=137
Oct 30 03:16:53.343800 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.343720    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" event={"ID":"345b910d-7a8d-4af2-bb0f-3b504c5df63e","Type":"ContainerDied","Data":"3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b"}
Oct 30 03:16:53.343800 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.343788    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz" event={"ID":"345b910d-7a8d-4af2-bb0f-3b504c5df63e","Type":"ContainerDied","Data":"72992b78c95beba0b86fb9b6e84d9810199c7e183c2efd78498dcec77841750d"}
Oct 30 03:16:53.350476 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.350461    4885 scope.go:122] "RemoveContainer" containerID="4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee"
Oct 30 03:16:53.350737 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:16:53.350718    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee\": container with ID starting with 4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee not found: ID does not exist" containerID="4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee"
Oct 30 03:16:53.350795 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.350745    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee"} err="failed to get container status \"4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee\": rpc error: code = NotFound desc = could not find container \"4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee\": container with ID starting with 4e2a19b5355626fb9481a9e2f4b2252d5130f31076c9d81acc93b2f4e1a9d5ee not found: ID does not exist"
Oct 30 03:16:53.350795 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.350760    4885 scope.go:122] "RemoveContainer" containerID="b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520"
Oct 30 03:16:53.359418 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.359404    4885 scope.go:122] "RemoveContainer" containerID="3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b"
Oct 30 03:16:53.366738 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.366720    4885 scope.go:122] "RemoveContainer" containerID="b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520"
Oct 30 03:16:53.367018 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:16:53.366998    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520\": container with ID starting with b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520 not found: ID does not exist" containerID="b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520"
Oct 30 03:16:53.367096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.367029    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520"} err="failed to get container status \"b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520\": rpc error: code = NotFound desc = could not find container \"b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520\": container with ID starting with b925ff8bbc99997039d5c605baef5af0b5782d037a95f1b86e6eb95efccfc520 not found: ID does not exist"
Oct 30 03:16:53.367096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.367051    4885 scope.go:122] "RemoveContainer" containerID="3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b"
Oct 30 03:16:53.367313 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:16:53.367297    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b\": container with ID starting with 3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b not found: ID does not exist" containerID="3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b"
Oct 30 03:16:53.367365 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.367321    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b"} err="failed to get container status \"3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b\": rpc error: code = NotFound desc = could not find container \"3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b\": container with ID starting with 3dfde2a961ab4beed2697c72b75caa7864b1d402f3512f833cd77ed821544b5b not found: ID does not exist"
Oct 30 03:16:53.405570 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.405550    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684\") pod \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") "
Oct 30 03:16:53.405701 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.405587    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-rbac-proxy-sar-config\") pod \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") "
Oct 30 03:16:53.405701 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.405655    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-api-access-2jkzh\" (UniqueName: \"kubernetes.io/projected/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-api-access-2jkzh\") pod \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") "
Oct 30 03:16:53.405823 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.405743    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/345b910d-7a8d-4af2-bb0f-3b504c5df63e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/345b910d-7a8d-4af2-bb0f-3b504c5df63e-proxy-tls\") pod \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\" (UID: \"345b910d-7a8d-4af2-bb0f-3b504c5df63e\") "
Oct 30 03:16:53.406025 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.405999    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-rbac-proxy-sar-config" pod "345b910d-7a8d-4af2-bb0f-3b504c5df63e" (UID: "345b910d-7a8d-4af2-bb0f-3b504c5df63e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:53.407925 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.407899    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-api-access-2jkzh" pod "345b910d-7a8d-4af2-bb0f-3b504c5df63e" (UID: "345b910d-7a8d-4af2-bb0f-3b504c5df63e"). InnerVolumeSpecName "kube-api-access-2jkzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:53.408024 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.408001    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/345b910d-7a8d-4af2-bb0f-3b504c5df63e-proxy-tls" pod "345b910d-7a8d-4af2-bb0f-3b504c5df63e" (UID: "345b910d-7a8d-4af2-bb0f-3b504c5df63e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:53.408501 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.408479    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684" pod "345b910d-7a8d-4af2-bb0f-3b504c5df63e" (UID: "345b910d-7a8d-4af2-bb0f-3b504c5df63e"). InnerVolumeSpecName "pvc-6d9df4c4-f698-4e8c-b72f-13f2e5364720". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:16:53.434052 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.434029    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"]
Oct 30 03:16:53.437785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.437767    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-sink-689dc4c4c-grrzn"]
Oct 30 03:16:53.506720 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.506695    4885 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6d9df4c4-f698-4e8c-b72f-13f2e5364720\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684\") on node \"ip-10-0-140-132.ec2.internal\" "
Oct 30 03:16:53.506720 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.506719    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:16:53.506864 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.506729    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jkzh\" (UniqueName: \"kubernetes.io/projected/345b910d-7a8d-4af2-bb0f-3b504c5df63e-kube-api-access-2jkzh\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:16:53.506864 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.506740    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/345b910d-7a8d-4af2-bb0f-3b504c5df63e-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:16:53.514107 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.514092    4885 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6d9df4c4-f698-4e8c-b72f-13f2e5364720" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684") on node "ip-10-0-140-132.ec2.internal"
Oct 30 03:16:53.607310 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:53.607247    4885 reconciler_common.go:299] "Volume detached for volume \"pvc-6d9df4c4-f698-4e8c-b72f-13f2e5364720\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aed59200ea718684\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"csi-2563f79b13ff69d913cdeed25c9ccae8535d0f2ec09b5da363f0ebb72246a545\""
Oct 30 03:16:54.349978 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.349952    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"
Oct 30 03:16:54.417484 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.417454    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"]
Oct 30 03:16:54.421293 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.421259    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-dd4100c6-predictor-bdfdcb7db-ls6dz"]
Oct 30 03:16:54.624108 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.624032    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" path="/var/lib/kubelet/pods/345b910d-7a8d-4af2-bb0f-3b504c5df63e/volumes"
Oct 30 03:16:54.624427 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.624412    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf021c4-972a-458b-ace0-143ec43ff698" path="/var/lib/kubelet/pods/ddf021c4-972a-458b-ace0-143ec43ff698/volumes"
Oct 30 03:16:54.674597 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.674568    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"]
Oct 30 03:16:54.674993 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.674975    4885 cpu_manager.go:414] "containerMap: removing container" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kube-rbac-proxy"
Oct 30 03:16:54.675096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.674994    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kube-rbac-proxy"
Oct 30 03:16:54.675096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.675019    4885 cpu_manager.go:414] "containerMap: removing container" podUID="ddf021c4-972a-458b-ace0-143ec43ff698" containerName="kserve-container"
Oct 30 03:16:54.675096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.675026    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf021c4-972a-458b-ace0-143ec43ff698" containerName="kserve-container"
Oct 30 03:16:54.675096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.675045    4885 cpu_manager.go:414] "containerMap: removing container" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kserve-container"
Oct 30 03:16:54.675096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.675053    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kserve-container"
Oct 30 03:16:54.675347 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.675153    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf021c4-972a-458b-ace0-143ec43ff698" containerName="kserve-container"
Oct 30 03:16:54.675347 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.675168    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kserve-container"
Oct 30 03:16:54.675347 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.675183    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="345b910d-7a8d-4af2-bb0f-3b504c5df63e" containerName="kube-rbac-proxy"
Oct 30 03:16:54.685689 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.685668    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"
Oct 30 03:16:54.698390 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.698325    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"]
Oct 30 03:16:54.700306 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.700246    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:16:54.700444 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.700367    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:16:54.700525 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.700491    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:16:54.817229 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.817199    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpq8\" (UniqueName: \"kubernetes.io/projected/d16ec6c9-c672-4172-9fe0-b1c6106ac970-kube-api-access-hwpq8\") pod \"proxy-routing-71305a63-sink-658898d86c-b6d2x\" (UID: \"d16ec6c9-c672-4172-9fe0-b1c6106ac970\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"
Oct 30 03:16:54.918635 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.918532    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpq8\" (UniqueName: \"kubernetes.io/projected/d16ec6c9-c672-4172-9fe0-b1c6106ac970-kube-api-access-hwpq8\") pod \"proxy-routing-71305a63-sink-658898d86c-b6d2x\" (UID: \"d16ec6c9-c672-4172-9fe0-b1c6106ac970\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"
Oct 30 03:16:54.942262 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:54.942235    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpq8\" (UniqueName: \"kubernetes.io/projected/d16ec6c9-c672-4172-9fe0-b1c6106ac970-kube-api-access-hwpq8\") pod \"proxy-routing-71305a63-sink-658898d86c-b6d2x\" (UID: \"d16ec6c9-c672-4172-9fe0-b1c6106ac970\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"
Oct 30 03:16:55.001069 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.001038    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"
Oct 30 03:16:55.297362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.296630    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"]
Oct 30 03:16:55.300937 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:16:55.300907    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ec6c9_c672_4172_9fe0_b1c6106ac970.slice/crio-99a391c50cf25bd168969182f86a521ae2b8b11ba360e63bff07adfede65aca9 WatchSource:0}: Error finding container 99a391c50cf25bd168969182f86a521ae2b8b11ba360e63bff07adfede65aca9: Status 404 returned error can't find the container with id 99a391c50cf25bd168969182f86a521ae2b8b11ba360e63bff07adfede65aca9
Oct 30 03:16:55.342791 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.342769    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:16:55.355189 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.355164    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x" event={"ID":"d16ec6c9-c672-4172-9fe0-b1c6106ac970","Type":"ContainerStarted","Data":"99a391c50cf25bd168969182f86a521ae2b8b11ba360e63bff07adfede65aca9"}
Oct 30 03:16:55.357070 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.357012    4885 generic.go:356] "Generic (PLEG): container finished" podID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerID="9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8" exitCode=0
Oct 30 03:16:55.357070 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.357047    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" event={"ID":"55acea28-3670-4eb8-81e4-8cadf898cf84","Type":"ContainerDied","Data":"9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8"}
Oct 30 03:16:55.357195 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.357081    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"
Oct 30 03:16:55.357195 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.357103    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7" event={"ID":"55acea28-3670-4eb8-81e4-8cadf898cf84","Type":"ContainerDied","Data":"cb58682cc380b73c27ec1f040eb6f707e3538bfc7862a2b09ee4bb998752b25b"}
Oct 30 03:16:55.357195 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.357143    4885 scope.go:122] "RemoveContainer" containerID="1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf"
Oct 30 03:16:55.366451 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.366431    4885 scope.go:122] "RemoveContainer" containerID="9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8"
Oct 30 03:16:55.373805 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.373788    4885 scope.go:122] "RemoveContainer" containerID="2c807b97b044de3932bd18e4a2e1cbdeeb0f27f6f3cca469e00f01098bceac0a"
Oct 30 03:16:55.382704 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.382686    4885 scope.go:122] "RemoveContainer" containerID="1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf"
Oct 30 03:16:55.383018 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:16:55.382994    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf\": container with ID starting with 1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf not found: ID does not exist" containerID="1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf"
Oct 30 03:16:55.383075 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.383030    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf"} err="failed to get container status \"1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf\": rpc error: code = NotFound desc = could not find container \"1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf\": container with ID starting with 1de87de44d2b88e5441af107269375ae7b011a6338a38be322ad585ab71db8bf not found: ID does not exist"
Oct 30 03:16:55.383075 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.383048    4885 scope.go:122] "RemoveContainer" containerID="9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8"
Oct 30 03:16:55.383334 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:16:55.383318    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8\": container with ID starting with 9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8 not found: ID does not exist" containerID="9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8"
Oct 30 03:16:55.383394 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.383339    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8"} err="failed to get container status \"9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8\": rpc error: code = NotFound desc = could not find container \"9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8\": container with ID starting with 9250c76a101ba97222ad12b54ee7d46e6f6857c22c9fbed407e4e647848df3e8 not found: ID does not exist"
Oct 30 03:16:55.383394 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.383354    4885 scope.go:122] "RemoveContainer" containerID="2c807b97b044de3932bd18e4a2e1cbdeeb0f27f6f3cca469e00f01098bceac0a"
Oct 30 03:16:55.383600 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:16:55.383584    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c807b97b044de3932bd18e4a2e1cbdeeb0f27f6f3cca469e00f01098bceac0a\": container with ID starting with 2c807b97b044de3932bd18e4a2e1cbdeeb0f27f6f3cca469e00f01098bceac0a not found: ID does not exist" containerID="2c807b97b044de3932bd18e4a2e1cbdeeb0f27f6f3cca469e00f01098bceac0a"
Oct 30 03:16:55.383656 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.383604    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c807b97b044de3932bd18e4a2e1cbdeeb0f27f6f3cca469e00f01098bceac0a"} err="failed to get container status \"2c807b97b044de3932bd18e4a2e1cbdeeb0f27f6f3cca469e00f01098bceac0a\": rpc error: code = NotFound desc = could not find container \"2c807b97b044de3932bd18e4a2e1cbdeeb0f27f6f3cca469e00f01098bceac0a\": container with ID starting with 2c807b97b044de3932bd18e4a2e1cbdeeb0f27f6f3cca469e00f01098bceac0a not found: ID does not exist"
Oct 30 03:16:55.423719 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.423690    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55acea28-3670-4eb8-81e4-8cadf898cf84-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55acea28-3670-4eb8-81e4-8cadf898cf84-kserve-provision-location\") pod \"55acea28-3670-4eb8-81e4-8cadf898cf84\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") "
Oct 30 03:16:55.423885 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.423745    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-rbac-proxy-sar-config\") pod \"55acea28-3670-4eb8-81e4-8cadf898cf84\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") "
Oct 30 03:16:55.423885 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.423799    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-api-access-wcbk2\" (UniqueName: \"kubernetes.io/projected/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-api-access-wcbk2\") pod \"55acea28-3670-4eb8-81e4-8cadf898cf84\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") "
Oct 30 03:16:55.423885 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.423846    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55acea28-3670-4eb8-81e4-8cadf898cf84-proxy-tls\" (UniqueName: \"kubernetes.io/secret/55acea28-3670-4eb8-81e4-8cadf898cf84-proxy-tls\") pod \"55acea28-3670-4eb8-81e4-8cadf898cf84\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") "
Oct 30 03:16:55.424004 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.423914    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-cabundle-cert\") pod \"55acea28-3670-4eb8-81e4-8cadf898cf84\" (UID: \"55acea28-3670-4eb8-81e4-8cadf898cf84\") "
Oct 30 03:16:55.424060 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.424040    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55acea28-3670-4eb8-81e4-8cadf898cf84-kserve-provision-location" pod "55acea28-3670-4eb8-81e4-8cadf898cf84" (UID: "55acea28-3670-4eb8-81e4-8cadf898cf84"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:55.424143 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.424123    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55acea28-3670-4eb8-81e4-8cadf898cf84-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:16:55.424359 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.424155    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-rbac-proxy-sar-config" pod "55acea28-3670-4eb8-81e4-8cadf898cf84" (UID: "55acea28-3670-4eb8-81e4-8cadf898cf84"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:55.424433 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.424368    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-cabundle-cert" pod "55acea28-3670-4eb8-81e4-8cadf898cf84" (UID: "55acea28-3670-4eb8-81e4-8cadf898cf84"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:55.426117 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.426091    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55acea28-3670-4eb8-81e4-8cadf898cf84-proxy-tls" pod "55acea28-3670-4eb8-81e4-8cadf898cf84" (UID: "55acea28-3670-4eb8-81e4-8cadf898cf84"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:55.426194 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.426095    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-api-access-wcbk2" pod "55acea28-3670-4eb8-81e4-8cadf898cf84" (UID: "55acea28-3670-4eb8-81e4-8cadf898cf84"). InnerVolumeSpecName "kube-api-access-wcbk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:55.525578 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.525501    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:16:55.525578 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.525528    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:16:55.525578 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.525538    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcbk2\" (UniqueName: \"kubernetes.io/projected/55acea28-3670-4eb8-81e4-8cadf898cf84-kube-api-access-wcbk2\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:16:55.525578 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.525547    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55acea28-3670-4eb8-81e4-8cadf898cf84-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:16:55.701155 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.701122    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"]
Oct 30 03:16:55.712976 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:55.712952    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-sklearn-v2-predictor-548cdc8969-469q7"]
Oct 30 03:16:55.922559 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:16:55.922485    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:16:56.368221 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:56.368182    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x" event={"ID":"d16ec6c9-c672-4172-9fe0-b1c6106ac970","Type":"ContainerStarted","Data":"8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de"}
Oct 30 03:16:56.368742 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:56.368282    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"
Oct 30 03:16:56.370950 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:56.370931    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"
Oct 30 03:16:56.407795 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:56.407754    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x" podStartSLOduration=2.407742443 podStartE2EDuration="2.407742443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:56.407177627 +0000 UTC m=+1321.917346664" watchObservedRunningTime="2026-10-30 03:16:56.407742443 +0000 UTC m=+1321.917911511"
Oct 30 03:16:56.624277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:56.624202    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" path="/var/lib/kubelet/pods/55acea28-3670-4eb8-81e4-8cadf898cf84/volumes"
Oct 30 03:16:58.354368 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.354332    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"]
Oct 30 03:16:58.354806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.354669    4885 cpu_manager.go:414] "containerMap: removing container" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kserve-container"
Oct 30 03:16:58.354806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.354681    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kserve-container"
Oct 30 03:16:58.354806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.354694    4885 cpu_manager.go:414] "containerMap: removing container" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kube-rbac-proxy"
Oct 30 03:16:58.354806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.354699    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kube-rbac-proxy"
Oct 30 03:16:58.354806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.354709    4885 cpu_manager.go:414] "containerMap: removing container" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="storage-initializer"
Oct 30 03:16:58.354806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.354714    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="storage-initializer"
Oct 30 03:16:58.354806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.354769    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kserve-container"
Oct 30 03:16:58.354806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.354777    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="55acea28-3670-4eb8-81e4-8cadf898cf84" containerName="kube-rbac-proxy"
Oct 30 03:16:58.359371 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.359350    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.374336 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.374318    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 30 03:16:58.374453 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.374413    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:16:58.385387 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.385359    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:16:58.385518 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.385416    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:16:58.385518 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.385500    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:16:58.385641 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.385573    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Oct 30 03:16:58.385702 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.385666    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:16:58.387478 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.387457    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"]
Oct 30 03:16:58.449402 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.449371    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.449551 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.449415    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.449551 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.449513    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.449551 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.449545    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.449690 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.449587    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht4f9\" (UniqueName: \"kubernetes.io/projected/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-api-access-ht4f9\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.550931 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.550900    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.551066 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.550944    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.551066 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.550966    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.551066 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.551012    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht4f9\" (UniqueName: \"kubernetes.io/projected/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-api-access-ht4f9\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.551066 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.551033    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.551487 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.551464    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.551761 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.551739    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.551815 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.551739    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.553567 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.553549    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.574541 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.574508    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht4f9\" (UniqueName: \"kubernetes.io/projected/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-api-access-ht4f9\") pod \"isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.676071 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.676002    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:16:58.981155 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:58.981126    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"]
Oct 30 03:16:58.984323 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:16:58.984294    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6937d6_2d0d_4931_b7d8_7bc4320b6917.slice/crio-809b3fa322ba5f8670bad64fa729af99507084234783abbb04344fb5fe4a85c3 WatchSource:0}: Error finding container 809b3fa322ba5f8670bad64fa729af99507084234783abbb04344fb5fe4a85c3: Status 404 returned error can't find the container with id 809b3fa322ba5f8670bad64fa729af99507084234783abbb04344fb5fe4a85c3
Oct 30 03:16:59.384219 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:59.384130    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" event={"ID":"7d6937d6-2d0d-4931-b7d8-7bc4320b6917","Type":"ContainerStarted","Data":"51a3aef5f1d47a6239260de3252eb95612ebc4adb0084a5d3399b0e3c98029f8"}
Oct 30 03:16:59.384219 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:16:59.384203    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" event={"ID":"7d6937d6-2d0d-4931-b7d8-7bc4320b6917","Type":"ContainerStarted","Data":"809b3fa322ba5f8670bad64fa729af99507084234783abbb04344fb5fe4a85c3"}
Oct 30 03:17:00.513814 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.513774    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"]
Oct 30 03:17:00.519278 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.519249    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:00.536021 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.536001    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-71305a63-kube-rbac-proxy-sar-config\""
Oct 30 03:17:00.536126 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.536045    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-71305a63-predictor-serving-cert\""
Oct 30 03:17:00.547919 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.547891    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-71305a63-dockercfg-zbtv4\""
Oct 30 03:17:00.549420 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.549403    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"]
Oct 30 03:17:00.669436 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.669402    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-rbac-proxy-sar-config\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:00.669436 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.669436    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shjxg\" (UniqueName: \"kubernetes.io/projected/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-api-access-shjxg\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:00.669667 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.669566    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a823b09-9fa5-428e-a559-3b00fdf2cc13-proxy-tls\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:00.770583 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.770500    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-rbac-proxy-sar-config\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:00.770583 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.770535    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shjxg\" (UniqueName: \"kubernetes.io/projected/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-api-access-shjxg\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:00.770831 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.770590    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a823b09-9fa5-428e-a559-3b00fdf2cc13-proxy-tls\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:00.771190 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.771164    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-rbac-proxy-sar-config\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:00.773076 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.773054    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a823b09-9fa5-428e-a559-3b00fdf2cc13-proxy-tls\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:00.794144 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:00.794120    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shjxg\" (UniqueName: \"kubernetes.io/projected/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-api-access-shjxg\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:03.402479 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:03.402445    4885 generic.go:356] "Generic (PLEG): container finished" podID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerID="51a3aef5f1d47a6239260de3252eb95612ebc4adb0084a5d3399b0e3c98029f8" exitCode=0
Oct 30 03:17:03.402897 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:03.402519    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" event={"ID":"7d6937d6-2d0d-4931-b7d8-7bc4320b6917","Type":"ContainerDied","Data":"51a3aef5f1d47a6239260de3252eb95612ebc4adb0084a5d3399b0e3c98029f8"}
Oct 30 03:17:04.408980 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:04.408951    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" event={"ID":"7d6937d6-2d0d-4931-b7d8-7bc4320b6917","Type":"ContainerStarted","Data":"c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34"}
Oct 30 03:17:04.409331 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:04.409009    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" event={"ID":"7d6937d6-2d0d-4931-b7d8-7bc4320b6917","Type":"ContainerStarted","Data":"538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174"}
Oct 30 03:17:04.409331 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:04.409210    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:17:04.454566 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:04.454515    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" podStartSLOduration=2.558627504 podStartE2EDuration="6.454498752s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.895871248s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:17:04.453030626 +0000 UTC m=+1329.963199654" watchObservedRunningTime="2026-10-30 03:17:04.454498752 +0000 UTC m=+1329.964667781"
Oct 30 03:17:05.413969 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:05.413943    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:17:07.323122 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:07.323083    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4451d851-709c-49df-aa2f-406fd7c534b7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:07.351773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:07.351738    4885 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4451d851-709c-49df-aa2f-406fd7c534b7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") device path: \"\"" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:07.423773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:07.423746    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4451d851-709c-49df-aa2f-406fd7c534b7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:07.423953 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:07.423889    4885 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4451d851-709c-49df-aa2f-406fd7c534b7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") DevicePath \"\"" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:07.438166 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:07.438138    4885 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4451d851-709c-49df-aa2f-406fd7c534b7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") DevicePath \"csi-025a7b7f6138860e088b813e39a7cc0736cfd3def06f05ec3ad519fdfd83dd29\"" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:07.617049 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:07.617015    4885 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4451d851-709c-49df-aa2f-406fd7c534b7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2301434e355f4f0bb5b465d72051d2ef14a7a357864b274bfb3d8e59f7281d43/globalmount\"" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:07.642135 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:07.642102    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4451d851-709c-49df-aa2f-406fd7c534b7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603\") pod \"proxy-routing-71305a63-predictor-cbc59d98b-phn9g\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") " pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:07.735739 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:07.735694    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:08.046758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:08.046734    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"]
Oct 30 03:17:08.049409 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:17:08.049380    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a823b09_9fa5_428e_a559_3b00fdf2cc13.slice/crio-fb56b04baf108d4dddd0826b0d17536b5dd0ddf99cb734c92ba0e6be3f369823 WatchSource:0}: Error finding container fb56b04baf108d4dddd0826b0d17536b5dd0ddf99cb734c92ba0e6be3f369823: Status 404 returned error can't find the container with id fb56b04baf108d4dddd0826b0d17536b5dd0ddf99cb734c92ba0e6be3f369823
Oct 30 03:17:08.429121 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:08.429035    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" event={"ID":"8a823b09-9fa5-428e-a559-3b00fdf2cc13","Type":"ContainerStarted","Data":"fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008"}
Oct 30 03:17:08.429121 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:08.429092    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" event={"ID":"8a823b09-9fa5-428e-a559-3b00fdf2cc13","Type":"ContainerStarted","Data":"ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d"}
Oct 30 03:17:08.429121 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:08.429116    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" event={"ID":"8a823b09-9fa5-428e-a559-3b00fdf2cc13","Type":"ContainerStarted","Data":"85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b"}
Oct 30 03:17:08.429640 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:08.429137    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" event={"ID":"8a823b09-9fa5-428e-a559-3b00fdf2cc13","Type":"ContainerStarted","Data":"fb56b04baf108d4dddd0826b0d17536b5dd0ddf99cb734c92ba0e6be3f369823"}
Oct 30 03:17:08.429640 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:08.429162    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:08.429640 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:08.429183    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:08.429640 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:08.429202    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:08.431746 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:08.431712    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:08.432173 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:08.432156    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:08.473646 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:08.473587    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podStartSLOduration=11.47357348 podStartE2EDuration="11.47357348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:17:08.472426456 +0000 UTC m=+1333.982595501" watchObservedRunningTime="2026-10-30 03:17:08.47357348 +0000 UTC m=+1333.983742505"
Oct 30 03:17:11.423149 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:11.423115    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:17:15.443096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:15.443068    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:17:25.921121 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:17:25.921082    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:17:41.424600 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:41.424551    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.54:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 03:17:47.542153 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:47.542120    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"]
Oct 30 03:17:47.542649 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:47.542319    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x" podUID="d16ec6c9-c672-4172-9fe0-b1c6106ac970" containerName="kserve-container" containerID="cri-o://8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de" gracePeriod=30
Oct 30 03:17:47.968024 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:47.967994    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"]
Oct 30 03:17:47.968299 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:47.968272    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kserve-container" containerID="cri-o://85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b" gracePeriod=30
Oct 30 03:17:47.968368 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:47.968296    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="agent" containerID="cri-o://fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008" gracePeriod=30
Oct 30 03:17:47.968368 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:47.968315    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kube-rbac-proxy" containerID="cri-o://ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d" gracePeriod=30
Oct 30 03:17:48.430362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:48.430268    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:17:48.620126 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:48.620092    4885 generic.go:356] "Generic (PLEG): container finished" podID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerID="ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d" exitCode=0
Oct 30 03:17:48.623356 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:48.623329    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" event={"ID":"8a823b09-9fa5-428e-a559-3b00fdf2cc13","Type":"ContainerDied","Data":"ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d"}
Oct 30 03:17:50.438806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:50.438764    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 30 03:17:51.424129 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:51.424094    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.54:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 03:17:55.439516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:55.439475    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 30 03:17:55.922262 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:17:55.922227    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:17:58.430571 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:17:58.430525    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:18:00.438878 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:00.438833    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 30 03:18:00.439289 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:00.438970    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:18:01.423890 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:01.423855    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.54:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 03:18:05.438908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:05.438867    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 30 03:18:08.429934 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:08.429842    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:18:08.430304 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:08.429996    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:18:10.438743 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:10.438706    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 30 03:18:11.936175 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:11.936148    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:18:15.439052 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:15.439013    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 30 03:18:17.606974 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:17.606947    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ec6c9_c672_4172_9fe0_b1c6106ac970.slice/crio-conmon-8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ec6c9_c672_4172_9fe0_b1c6106ac970.slice/crio-8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:18:17.607353 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:17.607035    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ec6c9_c672_4172_9fe0_b1c6106ac970.slice/crio-conmon-8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ec6c9_c672_4172_9fe0_b1c6106ac970.slice/crio-8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:18:17.607353 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:17.607097    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ec6c9_c672_4172_9fe0_b1c6106ac970.slice/crio-99a391c50cf25bd168969182f86a521ae2b8b11ba360e63bff07adfede65aca9\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ec6c9_c672_4172_9fe0_b1c6106ac970.slice/crio-8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ec6c9_c672_4172_9fe0_b1c6106ac970.slice/crio-conmon-8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:18:17.607469 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:17.607366    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ec6c9_c672_4172_9fe0_b1c6106ac970.slice/crio-conmon-8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ec6c9_c672_4172_9fe0_b1c6106ac970.slice/crio-8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:18:17.736629 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:17.736596    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"
Oct 30 03:18:17.747148 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:17.747125    4885 generic.go:356] "Generic (PLEG): container finished" podID="d16ec6c9-c672-4172-9fe0-b1c6106ac970" containerID="8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de" exitCode=137
Oct 30 03:18:17.747241 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:17.747163    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"
Oct 30 03:18:17.747241 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:17.747174    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x" event={"ID":"d16ec6c9-c672-4172-9fe0-b1c6106ac970","Type":"ContainerDied","Data":"8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de"}
Oct 30 03:18:17.747241 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:17.747209    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x" event={"ID":"d16ec6c9-c672-4172-9fe0-b1c6106ac970","Type":"ContainerDied","Data":"99a391c50cf25bd168969182f86a521ae2b8b11ba360e63bff07adfede65aca9"}
Oct 30 03:18:17.747241 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:17.747228    4885 scope.go:122] "RemoveContainer" containerID="8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de"
Oct 30 03:18:17.756624 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:17.756592    4885 scope.go:122] "RemoveContainer" containerID="8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de"
Oct 30 03:18:17.756930 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:17.756912    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de\": container with ID starting with 8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de not found: ID does not exist" containerID="8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de"
Oct 30 03:18:17.756989 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:17.756938    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de"} err="failed to get container status \"8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de\": rpc error: code = NotFound desc = could not find container \"8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de\": container with ID starting with 8283000a8c2d06227d2a529159bcc8987fb4a2d98b2bcee8854133f6b65ff9de not found: ID does not exist"
Oct 30 03:18:17.791890 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:17.791863    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16ec6c9-c672-4172-9fe0-b1c6106ac970-kube-api-access-hwpq8\" (UniqueName: \"kubernetes.io/projected/d16ec6c9-c672-4172-9fe0-b1c6106ac970-kube-api-access-hwpq8\") pod \"d16ec6c9-c672-4172-9fe0-b1c6106ac970\" (UID: \"d16ec6c9-c672-4172-9fe0-b1c6106ac970\") "
Oct 30 03:18:17.793969 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:17.793945    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16ec6c9-c672-4172-9fe0-b1c6106ac970-kube-api-access-hwpq8" pod "d16ec6c9-c672-4172-9fe0-b1c6106ac970" (UID: "d16ec6c9-c672-4172-9fe0-b1c6106ac970"). InnerVolumeSpecName "kube-api-access-hwpq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:17.892965 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:17.892935    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwpq8\" (UniqueName: \"kubernetes.io/projected/d16ec6c9-c672-4172-9fe0-b1c6106ac970-kube-api-access-hwpq8\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:18:18.023490 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:18.022605    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a823b09_9fa5_428e_a559_3b00fdf2cc13.slice/crio-fb56b04baf108d4dddd0826b0d17536b5dd0ddf99cb734c92ba0e6be3f369823\": RecentStats: unable to find data in memory cache]"
Oct 30 03:18:18.088646 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.088590    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"]
Oct 30 03:18:18.094874 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.091599    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-71305a63-sink-658898d86c-b6d2x"]
Oct 30 03:18:18.159785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.159755    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:18:18.196224 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.196197    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a823b09-9fa5-428e-a559-3b00fdf2cc13-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a823b09-9fa5-428e-a559-3b00fdf2cc13-proxy-tls\") pod \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") "
Oct 30 03:18:18.196355 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.196313    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603\") pod \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") "
Oct 30 03:18:18.196355 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.196342    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-rbac-proxy-sar-config\") pod \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") "
Oct 30 03:18:18.196423 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.196373    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-api-access-shjxg\" (UniqueName: \"kubernetes.io/projected/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-api-access-shjxg\") pod \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\" (UID: \"8a823b09-9fa5-428e-a559-3b00fdf2cc13\") "
Oct 30 03:18:18.196713 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.196690    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-rbac-proxy-sar-config" pod "8a823b09-9fa5-428e-a559-3b00fdf2cc13" (UID: "8a823b09-9fa5-428e-a559-3b00fdf2cc13"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:18.198431 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.198401    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a823b09-9fa5-428e-a559-3b00fdf2cc13-proxy-tls" pod "8a823b09-9fa5-428e-a559-3b00fdf2cc13" (UID: "8a823b09-9fa5-428e-a559-3b00fdf2cc13"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:18.198564 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.198544    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-api-access-shjxg" pod "8a823b09-9fa5-428e-a559-3b00fdf2cc13" (UID: "8a823b09-9fa5-428e-a559-3b00fdf2cc13"). InnerVolumeSpecName "kube-api-access-shjxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:18.199046 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.199029    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603" pod "8a823b09-9fa5-428e-a559-3b00fdf2cc13" (UID: "8a823b09-9fa5-428e-a559-3b00fdf2cc13"). InnerVolumeSpecName "pvc-4451d851-709c-49df-aa2f-406fd7c534b7". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:18:18.297321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.297288    4885 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4451d851-709c-49df-aa2f-406fd7c534b7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603\") on node \"ip-10-0-140-132.ec2.internal\" "
Oct 30 03:18:18.297321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.297315    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:18:18.297321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.297325    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shjxg\" (UniqueName: \"kubernetes.io/projected/8a823b09-9fa5-428e-a559-3b00fdf2cc13-kube-api-access-shjxg\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:18:18.297582 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.297336    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a823b09-9fa5-428e-a559-3b00fdf2cc13-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:18:18.307403 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.307359    4885 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4451d851-709c-49df-aa2f-406fd7c534b7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603") on node "ip-10-0-140-132.ec2.internal"
Oct 30 03:18:18.397793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.397767    4885 reconciler_common.go:299] "Volume detached for volume \"pvc-4451d851-709c-49df-aa2f-406fd7c534b7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0547f56e909a8d603\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"csi-025a7b7f6138860e088b813e39a7cc0736cfd3def06f05ec3ad519fdfd83dd29\""
Oct 30 03:18:18.441542 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.441510    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"]
Oct 30 03:18:18.441861 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.441830    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kserve-container" containerID="cri-o://538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174" gracePeriod=30
Oct 30 03:18:18.442083 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.441869    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kube-rbac-proxy" containerID="cri-o://c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34" gracePeriod=30
Oct 30 03:18:18.624174 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.624147    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16ec6c9-c672-4172-9fe0-b1c6106ac970" path="/var/lib/kubelet/pods/d16ec6c9-c672-4172-9fe0-b1c6106ac970/volumes"
Oct 30 03:18:18.753634 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.753575    4885 generic.go:356] "Generic (PLEG): container finished" podID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerID="fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008" exitCode=137
Oct 30 03:18:18.753634 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.753601    4885 generic.go:356] "Generic (PLEG): container finished" podID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerID="85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b" exitCode=137
Oct 30 03:18:18.753822 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.753657    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"
Oct 30 03:18:18.753822 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.753655    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" event={"ID":"8a823b09-9fa5-428e-a559-3b00fdf2cc13","Type":"ContainerDied","Data":"fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008"}
Oct 30 03:18:18.753822 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.753724    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" event={"ID":"8a823b09-9fa5-428e-a559-3b00fdf2cc13","Type":"ContainerDied","Data":"85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b"}
Oct 30 03:18:18.753822 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.753757    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g" event={"ID":"8a823b09-9fa5-428e-a559-3b00fdf2cc13","Type":"ContainerDied","Data":"fb56b04baf108d4dddd0826b0d17536b5dd0ddf99cb734c92ba0e6be3f369823"}
Oct 30 03:18:18.753822 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.753783    4885 scope.go:122] "RemoveContainer" containerID="fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008"
Oct 30 03:18:18.756176 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.756148    4885 generic.go:356] "Generic (PLEG): container finished" podID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerID="c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34" exitCode=0
Oct 30 03:18:18.756308 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.756191    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" event={"ID":"7d6937d6-2d0d-4931-b7d8-7bc4320b6917","Type":"ContainerDied","Data":"c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34"}
Oct 30 03:18:18.762106 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.762091    4885 scope.go:122] "RemoveContainer" containerID="ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d"
Oct 30 03:18:18.769870 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.769853    4885 scope.go:122] "RemoveContainer" containerID="85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b"
Oct 30 03:18:18.777275 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.777256    4885 scope.go:122] "RemoveContainer" containerID="fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008"
Oct 30 03:18:18.777527 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:18.777509    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008\": container with ID starting with fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008 not found: ID does not exist" containerID="fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008"
Oct 30 03:18:18.777587 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.777534    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008"} err="failed to get container status \"fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008\": rpc error: code = NotFound desc = could not find container \"fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008\": container with ID starting with fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008 not found: ID does not exist"
Oct 30 03:18:18.777587 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.777551    4885 scope.go:122] "RemoveContainer" containerID="ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d"
Oct 30 03:18:18.777815 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:18.777799    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d\": container with ID starting with ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d not found: ID does not exist" containerID="ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d"
Oct 30 03:18:18.777864 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.777818    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d"} err="failed to get container status \"ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d\": rpc error: code = NotFound desc = could not find container \"ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d\": container with ID starting with ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d not found: ID does not exist"
Oct 30 03:18:18.777864 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.777830    4885 scope.go:122] "RemoveContainer" containerID="85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b"
Oct 30 03:18:18.778040 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:18.778024    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b\": container with ID starting with 85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b not found: ID does not exist" containerID="85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b"
Oct 30 03:18:18.778078 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.778043    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b"} err="failed to get container status \"85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b\": rpc error: code = NotFound desc = could not find container \"85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b\": container with ID starting with 85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b not found: ID does not exist"
Oct 30 03:18:18.778078 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.778054    4885 scope.go:122] "RemoveContainer" containerID="fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008"
Oct 30 03:18:18.778237 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.778222    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008"} err="failed to get container status \"fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008\": rpc error: code = NotFound desc = could not find container \"fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008\": container with ID starting with fc2e3b9253d020d83bdedc4b2ae6cbd53d7554d889a910e907528c770fd7c008 not found: ID does not exist"
Oct 30 03:18:18.778280 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.778238    4885 scope.go:122] "RemoveContainer" containerID="ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d"
Oct 30 03:18:18.778406 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.778391    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d"} err="failed to get container status \"ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d\": rpc error: code = NotFound desc = could not find container \"ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d\": container with ID starting with ed8c5a77a7558d790dfa3c602b355041b06b4aad185ce515a90889d1bc43ad4d not found: ID does not exist"
Oct 30 03:18:18.778406 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.778405    4885 scope.go:122] "RemoveContainer" containerID="85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b"
Oct 30 03:18:18.778693 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.778603    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b"} err="failed to get container status \"85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b\": rpc error: code = NotFound desc = could not find container \"85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b\": container with ID starting with 85d57a8162429852f92fc0bf4ac2b309e1ba46c420af0d50f224ed5c4214086b not found: ID does not exist"
Oct 30 03:18:18.800490 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.800459    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"]
Oct 30 03:18:18.810812 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:18.810765    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-71305a63-predictor-cbc59d98b-phn9g"]
Oct 30 03:18:19.497233 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497199    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"]
Oct 30 03:18:19.497752 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497729    4885 cpu_manager.go:414] "containerMap: removing container" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kube-rbac-proxy"
Oct 30 03:18:19.497844 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497754    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kube-rbac-proxy"
Oct 30 03:18:19.497844 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497773    4885 cpu_manager.go:414] "containerMap: removing container" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="agent"
Oct 30 03:18:19.497844 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497781    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="agent"
Oct 30 03:18:19.497844 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497796    4885 cpu_manager.go:414] "containerMap: removing container" podUID="d16ec6c9-c672-4172-9fe0-b1c6106ac970" containerName="kserve-container"
Oct 30 03:18:19.497844 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497804    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16ec6c9-c672-4172-9fe0-b1c6106ac970" containerName="kserve-container"
Oct 30 03:18:19.497844 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497831    4885 cpu_manager.go:414] "containerMap: removing container" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kserve-container"
Oct 30 03:18:19.497844 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497837    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kserve-container"
Oct 30 03:18:19.498188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497924    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kube-rbac-proxy"
Oct 30 03:18:19.498188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497935    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="agent"
Oct 30 03:18:19.498188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497945    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16ec6c9-c672-4172-9fe0-b1c6106ac970" containerName="kserve-container"
Oct 30 03:18:19.498188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.497957    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" containerName="kserve-container"
Oct 30 03:18:19.502885 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.502865    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"
Oct 30 03:18:19.520536 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.520511    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"]
Oct 30 03:18:19.528937 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.528910    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:18:19.529052 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.528938    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:18:19.529102 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.529060    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:18:19.606927 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.606893    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jn8d\" (UniqueName: \"kubernetes.io/projected/463a783b-7aaf-4e2c-80d4-ab7b0883844e-kube-api-access-8jn8d\") pod \"proxy-routing-21889b45-sink-745bfdcbfd-qz9pz\" (UID: \"463a783b-7aaf-4e2c-80d4-ab7b0883844e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"
Oct 30 03:18:19.707813 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.707770    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jn8d\" (UniqueName: \"kubernetes.io/projected/463a783b-7aaf-4e2c-80d4-ab7b0883844e-kube-api-access-8jn8d\") pod \"proxy-routing-21889b45-sink-745bfdcbfd-qz9pz\" (UID: \"463a783b-7aaf-4e2c-80d4-ab7b0883844e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"
Oct 30 03:18:19.729756 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.729730    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jn8d\" (UniqueName: \"kubernetes.io/projected/463a783b-7aaf-4e2c-80d4-ab7b0883844e-kube-api-access-8jn8d\") pod \"proxy-routing-21889b45-sink-745bfdcbfd-qz9pz\" (UID: \"463a783b-7aaf-4e2c-80d4-ab7b0883844e\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"
Oct 30 03:18:19.815405 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:19.815337    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"
Oct 30 03:18:20.107026 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:20.106791    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"]
Oct 30 03:18:20.109237 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:18:20.109206    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463a783b_7aaf_4e2c_80d4_ab7b0883844e.slice/crio-605e4cff2caf18b107e304536d4bef70547919d843bf8361f3b83cbc360a240c WatchSource:0}: Error finding container 605e4cff2caf18b107e304536d4bef70547919d843bf8361f3b83cbc360a240c: Status 404 returned error can't find the container with id 605e4cff2caf18b107e304536d4bef70547919d843bf8361f3b83cbc360a240c
Oct 30 03:18:20.624336 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:20.624303    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a823b09-9fa5-428e-a559-3b00fdf2cc13" path="/var/lib/kubelet/pods/8a823b09-9fa5-428e-a559-3b00fdf2cc13/volumes"
Oct 30 03:18:20.766986 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:20.766950    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz" event={"ID":"463a783b-7aaf-4e2c-80d4-ab7b0883844e","Type":"ContainerStarted","Data":"2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569"}
Oct 30 03:18:20.767363 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:20.767005    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz" event={"ID":"463a783b-7aaf-4e2c-80d4-ab7b0883844e","Type":"ContainerStarted","Data":"605e4cff2caf18b107e304536d4bef70547919d843bf8361f3b83cbc360a240c"}
Oct 30 03:18:20.767363 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:20.767199    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"
Oct 30 03:18:20.769460 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:20.769443    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"
Oct 30 03:18:20.804070 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:20.804026    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz" podStartSLOduration=1.8040114680000001 podStartE2EDuration="1.804011468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:20.802967481 +0000 UTC m=+1406.313136501" watchObservedRunningTime="2026-10-30 03:18:20.804011468 +0000 UTC m=+1406.314180515"
Oct 30 03:18:21.419443 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:21.419409    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Oct 30 03:18:21.424028 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:21.424003    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.54:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.54:8080: connect: connection refused"
Oct 30 03:18:22.990899 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:22.990874    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:18:23.035486 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.035462    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-cabundle-cert\") pod \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") "
Oct 30 03:18:23.035600 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.035504    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kserve-provision-location\") pod \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") "
Oct 30 03:18:23.035600 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.035522    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-api-access-ht4f9\" (UniqueName: \"kubernetes.io/projected/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-api-access-ht4f9\") pod \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") "
Oct 30 03:18:23.035600 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.035578    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-proxy-tls\") pod \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") "
Oct 30 03:18:23.035825 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.035632    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-rbac-proxy-sar-config\") pod \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\" (UID: \"7d6937d6-2d0d-4931-b7d8-7bc4320b6917\") "
Oct 30 03:18:23.035888 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.035840    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-cabundle-cert" pod "7d6937d6-2d0d-4931-b7d8-7bc4320b6917" (UID: "7d6937d6-2d0d-4931-b7d8-7bc4320b6917"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:23.035888 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.035837    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kserve-provision-location" pod "7d6937d6-2d0d-4931-b7d8-7bc4320b6917" (UID: "7d6937d6-2d0d-4931-b7d8-7bc4320b6917"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:23.036081 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.036060    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-rbac-proxy-sar-config" pod "7d6937d6-2d0d-4931-b7d8-7bc4320b6917" (UID: "7d6937d6-2d0d-4931-b7d8-7bc4320b6917"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:23.037778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.037759    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-proxy-tls" pod "7d6937d6-2d0d-4931-b7d8-7bc4320b6917" (UID: "7d6937d6-2d0d-4931-b7d8-7bc4320b6917"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:23.037860 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.037814    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-api-access-ht4f9" pod "7d6937d6-2d0d-4931-b7d8-7bc4320b6917" (UID: "7d6937d6-2d0d-4931-b7d8-7bc4320b6917"). InnerVolumeSpecName "kube-api-access-ht4f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:23.136706 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.136630    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.136706 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.136658    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.136706 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.136668    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.136706 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.136677    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.136706 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.136686    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht4f9\" (UniqueName: \"kubernetes.io/projected/7d6937d6-2d0d-4931-b7d8-7bc4320b6917-kube-api-access-ht4f9\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.781504 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.781473    4885 generic.go:356] "Generic (PLEG): container finished" podID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerID="538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174" exitCode=0
Oct 30 03:18:23.781693 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.781526    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"
Oct 30 03:18:23.781693 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.781551    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" event={"ID":"7d6937d6-2d0d-4931-b7d8-7bc4320b6917","Type":"ContainerDied","Data":"538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174"}
Oct 30 03:18:23.781693 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.781629    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd" event={"ID":"7d6937d6-2d0d-4931-b7d8-7bc4320b6917","Type":"ContainerDied","Data":"809b3fa322ba5f8670bad64fa729af99507084234783abbb04344fb5fe4a85c3"}
Oct 30 03:18:23.781693 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.781637    4885 scope.go:122] "RemoveContainer" containerID="c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34"
Oct 30 03:18:23.790448 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.790435    4885 scope.go:122] "RemoveContainer" containerID="538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174"
Oct 30 03:18:23.797725 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.797705    4885 scope.go:122] "RemoveContainer" containerID="51a3aef5f1d47a6239260de3252eb95612ebc4adb0084a5d3399b0e3c98029f8"
Oct 30 03:18:23.805049 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.805035    4885 scope.go:122] "RemoveContainer" containerID="c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34"
Oct 30 03:18:23.805316 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:23.805297    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34\": container with ID starting with c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34 not found: ID does not exist" containerID="c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34"
Oct 30 03:18:23.805357 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.805324    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34"} err="failed to get container status \"c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34\": rpc error: code = NotFound desc = could not find container \"c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34\": container with ID starting with c7e859205550ea8ad6a4c1aadcfbc7e377ce32556e4aed4155cdaaa2c5f13c34 not found: ID does not exist"
Oct 30 03:18:23.805357 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.805340    4885 scope.go:122] "RemoveContainer" containerID="538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174"
Oct 30 03:18:23.805570 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:23.805553    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174\": container with ID starting with 538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174 not found: ID does not exist" containerID="538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174"
Oct 30 03:18:23.805625 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.805577    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174"} err="failed to get container status \"538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174\": rpc error: code = NotFound desc = could not find container \"538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174\": container with ID starting with 538e0ec22bda64c914c4eb5c59b5d2098e38153eb84e16af7250e0c451945174 not found: ID does not exist"
Oct 30 03:18:23.805625 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.805593    4885 scope.go:122] "RemoveContainer" containerID="51a3aef5f1d47a6239260de3252eb95612ebc4adb0084a5d3399b0e3c98029f8"
Oct 30 03:18:23.805861 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:23.805844    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a3aef5f1d47a6239260de3252eb95612ebc4adb0084a5d3399b0e3c98029f8\": container with ID starting with 51a3aef5f1d47a6239260de3252eb95612ebc4adb0084a5d3399b0e3c98029f8 not found: ID does not exist" containerID="51a3aef5f1d47a6239260de3252eb95612ebc4adb0084a5d3399b0e3c98029f8"
Oct 30 03:18:23.805908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.805867    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a3aef5f1d47a6239260de3252eb95612ebc4adb0084a5d3399b0e3c98029f8"} err="failed to get container status \"51a3aef5f1d47a6239260de3252eb95612ebc4adb0084a5d3399b0e3c98029f8\": rpc error: code = NotFound desc = could not find container \"51a3aef5f1d47a6239260de3252eb95612ebc4adb0084a5d3399b0e3c98029f8\": container with ID starting with 51a3aef5f1d47a6239260de3252eb95612ebc4adb0084a5d3399b0e3c98029f8 not found: ID does not exist"
Oct 30 03:18:23.837060 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.837034    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"]
Oct 30 03:18:23.847109 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:23.847073    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-xgboost-v2-predictor-57fb4748c4-kzhqd"]
Oct 30 03:18:24.624258 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:24.624229    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" path="/var/lib/kubelet/pods/7d6937d6-2d0d-4931-b7d8-7bc4320b6917/volumes"
Oct 30 03:18:25.152736 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.152704    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"]
Oct 30 03:18:25.153097 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.153078    4885 cpu_manager.go:414] "containerMap: removing container" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kube-rbac-proxy"
Oct 30 03:18:25.153191 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.153099    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kube-rbac-proxy"
Oct 30 03:18:25.153191 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.153116    4885 cpu_manager.go:414] "containerMap: removing container" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kserve-container"
Oct 30 03:18:25.153191 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.153123    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kserve-container"
Oct 30 03:18:25.153191 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.153136    4885 cpu_manager.go:414] "containerMap: removing container" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="storage-initializer"
Oct 30 03:18:25.153191 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.153144    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="storage-initializer"
Oct 30 03:18:25.153438 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.153230    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kube-rbac-proxy"
Oct 30 03:18:25.153438 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.153241    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6937d6-2d0d-4931-b7d8-7bc4320b6917" containerName="kserve-container"
Oct 30 03:18:25.157841 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.157824    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:25.171593 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.171573    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-21889b45-dockercfg-nfpc6\""
Oct 30 03:18:25.181373 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.181351    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-21889b45-predictor-serving-cert\""
Oct 30 03:18:25.181373 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.181361    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-21889b45-kube-rbac-proxy-sar-config\""
Oct 30 03:18:25.182893 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.182873    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"]
Oct 30 03:18:25.253648 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.253606    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403a5576-f3f2-46d8-9ba0-2fb0584486ad-proxy-tls\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:25.253793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.253711    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-rbac-proxy-sar-config\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:25.253793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.253776    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw7l6\" (UniqueName: \"kubernetes.io/projected/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-api-access-jw7l6\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:25.354592 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.354558    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403a5576-f3f2-46d8-9ba0-2fb0584486ad-proxy-tls\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:25.354778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.354629    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-rbac-proxy-sar-config\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:25.354778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.354654    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw7l6\" (UniqueName: \"kubernetes.io/projected/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-api-access-jw7l6\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:25.355345 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.355315    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-rbac-proxy-sar-config\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:25.357156 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.357122    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403a5576-f3f2-46d8-9ba0-2fb0584486ad-proxy-tls\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:25.376248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:25.376225    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw7l6\" (UniqueName: \"kubernetes.io/projected/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-api-access-jw7l6\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:25.921909 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:25.921875    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:18:26.186730 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.186655    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"]
Oct 30 03:18:26.191512 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.191491    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.205842 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.205822    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:18:26.205928 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.205898    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:18:26.215806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.215787    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:18:26.215806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.215783    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Oct 30 03:18:26.215984 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.215855    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:18:26.215984 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.215873    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Oct 30 03:18:26.215984 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.215873    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:18:26.217467 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.217447    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"]
Oct 30 03:18:26.261618 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.261584    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.261719 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.261652    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.261778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.261759    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlq4c\" (UniqueName: \"kubernetes.io/projected/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-api-access-xlq4c\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.261812 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.261796    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.261854 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.261835    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d8fa08f-3d4d-446b-b890-518c7a00dd29-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.362290 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.362264    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d8fa08f-3d4d-446b-b890-518c7a00dd29-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.362395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.362310    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.362395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.362333    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.362395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.362364    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlq4c\" (UniqueName: \"kubernetes.io/projected/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-api-access-xlq4c\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.362529 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.362397    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.362769 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.362733    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.362978 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.362958    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.363132 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.363100    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.364970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.364952    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d8fa08f-3d4d-446b-b890-518c7a00dd29-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.383699 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.383677    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlq4c\" (UniqueName: \"kubernetes.io/projected/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-api-access-xlq4c\") pod \"isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") " pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.509161 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.509126    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:26.815961 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:26.814340    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"]
Oct 30 03:18:26.821211 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:18:26.821179    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8fa08f_3d4d_446b_b890_518c7a00dd29.slice/crio-e980c01bf68aa010fcc8f756ed4fe64f367eb3d575a2d51f3572486656d81bda WatchSource:0}: Error finding container e980c01bf68aa010fcc8f756ed4fe64f367eb3d575a2d51f3572486656d81bda: Status 404 returned error can't find the container with id e980c01bf68aa010fcc8f756ed4fe64f367eb3d575a2d51f3572486656d81bda
Oct 30 03:18:27.801370 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:27.801334    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr" event={"ID":"3d8fa08f-3d4d-446b-b890-518c7a00dd29","Type":"ContainerStarted","Data":"a0f082df6ed596928ad2bafef5ed012e95c67693ea848491b24d51d5872faba6"}
Oct 30 03:18:27.801806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:27.801395    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr" event={"ID":"3d8fa08f-3d4d-446b-b890-518c7a00dd29","Type":"ContainerStarted","Data":"e980c01bf68aa010fcc8f756ed4fe64f367eb3d575a2d51f3572486656d81bda"}
Oct 30 03:18:30.815360 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:30.815330    4885 generic.go:356] "Generic (PLEG): container finished" podID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerID="a0f082df6ed596928ad2bafef5ed012e95c67693ea848491b24d51d5872faba6" exitCode=0
Oct 30 03:18:30.815824 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:30.815382    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr" event={"ID":"3d8fa08f-3d4d-446b-b890-518c7a00dd29","Type":"ContainerDied","Data":"a0f082df6ed596928ad2bafef5ed012e95c67693ea848491b24d51d5872faba6"}
Oct 30 03:18:31.822018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:31.821984    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr" event={"ID":"3d8fa08f-3d4d-446b-b890-518c7a00dd29","Type":"ContainerStarted","Data":"707de324d7f53ec62dcc5cf025560cb0471372bfa4d2368156fad87a0b046bb1"}
Oct 30 03:18:31.822425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:31.822051    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr" event={"ID":"3d8fa08f-3d4d-446b-b890-518c7a00dd29","Type":"ContainerStarted","Data":"8d2f1540baf35f9a87d8b0afc1d9d8f0c199f2b1ee588ccb3fe2006c94bd01a4"}
Oct 30 03:18:31.822425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:31.822263    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:31.875459 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:31.875399    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr" podStartSLOduration=2.338663759 podStartE2EDuration="5.875380848s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.536717089s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:31.874283388 +0000 UTC m=+1417.384452438" watchObservedRunningTime="2026-10-30 03:18:31.875380848 +0000 UTC m=+1417.385549872"
Oct 30 03:18:32.827143 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:32.827106    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:35.543785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:35.543752    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:35.568918 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:35.568879    4885 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") device path: \"\"" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:35.644906 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:35.644864    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:35.645099 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:35.645004    4885 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") DevicePath \"\"" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:35.657260 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:35.657233    4885 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") DevicePath \"csi-90cd0c3d0c6e9d2561aa4eeb9e4df42f8a45df9c3822250dcd8b46f166be07ec\"" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:35.851408 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:35.851369    4885 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7be85bc228edde09f84fc9146e78bfe54232f53eeb4274891255944ee0c0aaa5/globalmount\"" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:35.874298 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:35.874271    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:35.971497 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:35.971456    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:36.277890 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:36.277856    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"]
Oct 30 03:18:36.281918 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:18:36.281882    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403a5576_f3f2_46d8_9ba0_2fb0584486ad.slice/crio-068f3919e8f202cb0069b3a418424331aa57eba8adab8293883ffe1406d89d7e WatchSource:0}: Error finding container 068f3919e8f202cb0069b3a418424331aa57eba8adab8293883ffe1406d89d7e: Status 404 returned error can't find the container with id 068f3919e8f202cb0069b3a418424331aa57eba8adab8293883ffe1406d89d7e
Oct 30 03:18:36.859289 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:36.859250    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" event={"ID":"403a5576-f3f2-46d8-9ba0-2fb0584486ad","Type":"ContainerStarted","Data":"ef49d10c05dff91efd775b3429ebcd2480ce812e9f7ef33b110e343caeff6243"}
Oct 30 03:18:36.859728 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:36.859303    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" event={"ID":"403a5576-f3f2-46d8-9ba0-2fb0584486ad","Type":"ContainerStarted","Data":"4e577457a665e2ce44603d7a4addd386dfb1deb33553e521cbbfc4d7dfee5c1f"}
Oct 30 03:18:36.859728 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:36.859323    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" event={"ID":"403a5576-f3f2-46d8-9ba0-2fb0584486ad","Type":"ContainerStarted","Data":"068f3919e8f202cb0069b3a418424331aa57eba8adab8293883ffe1406d89d7e"}
Oct 30 03:18:36.859728 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:36.859390    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:36.900556 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:36.900506    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" podStartSLOduration=14.900487673 podStartE2EDuration="14.900487673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:36.900273154 +0000 UTC m=+1422.410442180" watchObservedRunningTime="2026-10-30 03:18:36.900487673 +0000 UTC m=+1422.410656699"
Oct 30 03:18:37.863322 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:37.863286    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:37.865524 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:37.865501    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:38.839004 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:38.838977    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:18:44.877777 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:44.877749    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:18:52.331953 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.331920    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"]
Oct 30 03:18:52.336023 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.336003    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.361025 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.361003    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"]
Oct 30 03:18:52.384965 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.384938    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.385067 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.384973    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcv7p\" (UniqueName: \"kubernetes.io/projected/f61e65a3-bf78-475e-831c-65756f196c23-kube-api-access-dcv7p\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.385104 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.385078    4885 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") DevicePath \"csi-90cd0c3d0c6e9d2561aa4eeb9e4df42f8a45df9c3822250dcd8b46f166be07ec\"" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.385104 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.385093    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f61e65a3-bf78-475e-831c-65756f196c23-kube-rbac-proxy-sar-config\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.385206 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.385147    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f61e65a3-bf78-475e-831c-65756f196c23-proxy-tls\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.397173 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.397154    4885 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") DevicePath \"csi-90cd0c3d0c6e9d2561aa4eeb9e4df42f8a45df9c3822250dcd8b46f166be07ec\"" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.418315 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.418295    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.486051 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.486026    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcv7p\" (UniqueName: \"kubernetes.io/projected/f61e65a3-bf78-475e-831c-65756f196c23-kube-api-access-dcv7p\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.486176 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.486062    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f61e65a3-bf78-475e-831c-65756f196c23-kube-rbac-proxy-sar-config\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.486176 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.486092    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f61e65a3-bf78-475e-831c-65756f196c23-proxy-tls\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.486672 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.486652    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f61e65a3-bf78-475e-831c-65756f196c23-kube-rbac-proxy-sar-config\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.488470 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.488455    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f61e65a3-bf78-475e-831c-65756f196c23-proxy-tls\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.507939 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.507912    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcv7p\" (UniqueName: \"kubernetes.io/projected/f61e65a3-bf78-475e-831c-65756f196c23-kube-api-access-dcv7p\") pod \"proxy-routing-21889b45-predictor-678db9d655-g62rn\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") " pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.650256 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.650194    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:52.961819 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:52.961785    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"]
Oct 30 03:18:52.966510 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:18:52.966483    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61e65a3_bf78_475e_831c_65756f196c23.slice/crio-8c81aeacad2eb2c18e0e751dd7cc2ef2b87f747ef2103d77d1eaa0a5d1b42298 WatchSource:0}: Error finding container 8c81aeacad2eb2c18e0e751dd7cc2ef2b87f747ef2103d77d1eaa0a5d1b42298: Status 404 returned error can't find the container with id 8c81aeacad2eb2c18e0e751dd7cc2ef2b87f747ef2103d77d1eaa0a5d1b42298
Oct 30 03:18:53.935367 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:53.935338    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" event={"ID":"f61e65a3-bf78-475e-831c-65756f196c23","Type":"ContainerStarted","Data":"a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9"}
Oct 30 03:18:53.935882 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:53.935390    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" event={"ID":"f61e65a3-bf78-475e-831c-65756f196c23","Type":"ContainerStarted","Data":"2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb"}
Oct 30 03:18:53.935882 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:53.935411    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" event={"ID":"f61e65a3-bf78-475e-831c-65756f196c23","Type":"ContainerStarted","Data":"8c81aeacad2eb2c18e0e751dd7cc2ef2b87f747ef2103d77d1eaa0a5d1b42298"}
Oct 30 03:18:53.935882 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:53.935433    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:53.984633 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:53.984573    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" podStartSLOduration=1.984559974 podStartE2EDuration="1.984559974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:53.974033259 +0000 UTC m=+1439.484202312" watchObservedRunningTime="2026-10-30 03:18:53.984559974 +0000 UTC m=+1439.494729000"
Oct 30 03:18:54.939381 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:54.939356    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:54.941482 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:18:54.941460    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:18:55.921595 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:18:55.921561    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:19:01.951950 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:01.951923    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:19:02.034753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:02.034723    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"]
Oct 30 03:19:02.035052 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:02.035013    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kserve-container" containerID="cri-o://4e577457a665e2ce44603d7a4addd386dfb1deb33553e521cbbfc4d7dfee5c1f" gracePeriod=30
Oct 30 03:19:02.035052 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:02.035020    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kube-rbac-proxy" containerID="cri-o://ef49d10c05dff91efd775b3429ebcd2480ce812e9f7ef33b110e343caeff6243" gracePeriod=30
Oct 30 03:19:02.973803 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:02.973768    4885 generic.go:356] "Generic (PLEG): container finished" podID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerID="ef49d10c05dff91efd775b3429ebcd2480ce812e9f7ef33b110e343caeff6243" exitCode=0
Oct 30 03:19:02.974191 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:02.973842    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" event={"ID":"403a5576-f3f2-46d8-9ba0-2fb0584486ad","Type":"ContainerDied","Data":"ef49d10c05dff91efd775b3429ebcd2480ce812e9f7ef33b110e343caeff6243"}
Oct 30 03:19:04.873674 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:04.873636    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 30 03:19:08.898927 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:08.898896    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:19:09.873431 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:09.873388    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 30 03:19:14.873768 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:14.873736    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 30 03:19:14.874117 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:14.873854    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:19:16.398483 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:16.398450    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"]
Oct 30 03:19:16.399001 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:16.398737    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr" podUID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerName="kserve-container" containerID="cri-o://8d2f1540baf35f9a87d8b0afc1d9d8f0c199f2b1ee588ccb3fe2006c94bd01a4" gracePeriod=30
Oct 30 03:19:16.399001 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:16.398797    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr" podUID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerName="kube-rbac-proxy" containerID="cri-o://707de324d7f53ec62dcc5cf025560cb0471372bfa4d2368156fad87a0b046bb1" gracePeriod=30
Oct 30 03:19:17.040520 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.040492    4885 generic.go:356] "Generic (PLEG): container finished" podID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerID="707de324d7f53ec62dcc5cf025560cb0471372bfa4d2368156fad87a0b046bb1" exitCode=0
Oct 30 03:19:17.040520 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.040517    4885 generic.go:356] "Generic (PLEG): container finished" podID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerID="8d2f1540baf35f9a87d8b0afc1d9d8f0c199f2b1ee588ccb3fe2006c94bd01a4" exitCode=0
Oct 30 03:19:17.040732 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.040556    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr" event={"ID":"3d8fa08f-3d4d-446b-b890-518c7a00dd29","Type":"ContainerDied","Data":"707de324d7f53ec62dcc5cf025560cb0471372bfa4d2368156fad87a0b046bb1"}
Oct 30 03:19:17.040732 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.040640    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr" event={"ID":"3d8fa08f-3d4d-446b-b890-518c7a00dd29","Type":"ContainerDied","Data":"8d2f1540baf35f9a87d8b0afc1d9d8f0c199f2b1ee588ccb3fe2006c94bd01a4"}
Oct 30 03:19:17.040732 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.040665    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr" event={"ID":"3d8fa08f-3d4d-446b-b890-518c7a00dd29","Type":"ContainerDied","Data":"e980c01bf68aa010fcc8f756ed4fe64f367eb3d575a2d51f3572486656d81bda"}
Oct 30 03:19:17.040732 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.040693    4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e980c01bf68aa010fcc8f756ed4fe64f367eb3d575a2d51f3572486656d81bda"
Oct 30 03:19:17.043970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.043952    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:19:17.105704 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.105608    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-cabundle-cert\") pod \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") "
Oct 30 03:19:17.105852 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.105708    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d8fa08f-3d4d-446b-b890-518c7a00dd29-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d8fa08f-3d4d-446b-b890-518c7a00dd29-proxy-tls\") pod \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") "
Oct 30 03:19:17.105852 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.105762    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kserve-provision-location\") pod \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") "
Oct 30 03:19:17.105852 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.105820    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-rbac-proxy-sar-config\") pod \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") "
Oct 30 03:19:17.105979 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.105870    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-api-access-xlq4c\" (UniqueName: \"kubernetes.io/projected/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-api-access-xlq4c\") pod \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\" (UID: \"3d8fa08f-3d4d-446b-b890-518c7a00dd29\") "
Oct 30 03:19:17.105979 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.105929    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-cabundle-cert" pod "3d8fa08f-3d4d-446b-b890-518c7a00dd29" (UID: "3d8fa08f-3d4d-446b-b890-518c7a00dd29"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:17.106140 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.106122    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:19:17.106210 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.106140    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kserve-provision-location" pod "3d8fa08f-3d4d-446b-b890-518c7a00dd29" (UID: "3d8fa08f-3d4d-446b-b890-518c7a00dd29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:17.106273 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.106200    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-rbac-proxy-sar-config" pod "3d8fa08f-3d4d-446b-b890-518c7a00dd29" (UID: "3d8fa08f-3d4d-446b-b890-518c7a00dd29"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:17.107873 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.107847    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8fa08f-3d4d-446b-b890-518c7a00dd29-proxy-tls" pod "3d8fa08f-3d4d-446b-b890-518c7a00dd29" (UID: "3d8fa08f-3d4d-446b-b890-518c7a00dd29"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:17.108099 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.108083    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-api-access-xlq4c" pod "3d8fa08f-3d4d-446b-b890-518c7a00dd29" (UID: "3d8fa08f-3d4d-446b-b890-518c7a00dd29"). InnerVolumeSpecName "kube-api-access-xlq4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:17.207420 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.207399    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:19:17.207420 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.207419    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlq4c\" (UniqueName: \"kubernetes.io/projected/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kube-api-access-xlq4c\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:19:17.207573 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.207429    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d8fa08f-3d4d-446b-b890-518c7a00dd29-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:19:17.207573 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:17.207438    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d8fa08f-3d4d-446b-b890-518c7a00dd29-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:19:18.044353 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:18.044326    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"
Oct 30 03:19:18.101552 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:18.101512    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"]
Oct 30 03:19:18.112096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:18.112066    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-predictive-lightgbm-v2-predictor-5ffc8686c5-4kbhr"]
Oct 30 03:19:18.624181 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:18.624153    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" path="/var/lib/kubelet/pods/3d8fa08f-3d4d-446b-b890-518c7a00dd29/volumes"
Oct 30 03:19:19.873456 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:19.873408    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 30 03:19:24.873385 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:24.873340    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 30 03:19:25.921778 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:19:25.921742    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:19:29.873804 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:29.873761    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 30 03:19:30.967707 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.967667    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"]
Oct 30 03:19:30.968215 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.968180    4885 cpu_manager.go:414] "containerMap: removing container" podUID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerName="kserve-container"
Oct 30 03:19:30.968215 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.968198    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerName="kserve-container"
Oct 30 03:19:30.968338 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.968231    4885 cpu_manager.go:414] "containerMap: removing container" podUID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerName="kube-rbac-proxy"
Oct 30 03:19:30.968338 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.968239    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerName="kube-rbac-proxy"
Oct 30 03:19:30.968338 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.968254    4885 cpu_manager.go:414] "containerMap: removing container" podUID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerName="storage-initializer"
Oct 30 03:19:30.968338 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.968262    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerName="storage-initializer"
Oct 30 03:19:30.968540 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.968342    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerName="kserve-container"
Oct 30 03:19:30.968540 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.968361    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8fa08f-3d4d-446b-b890-518c7a00dd29" containerName="kube-rbac-proxy"
Oct 30 03:19:30.973235 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.973212    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:30.987843 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.987820    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:19:30.987843 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.987821    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:19:30.997816 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.997791    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-predictor-serving-cert\""
Oct 30 03:19:30.997816 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.997814    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Oct 30 03:19:30.997992 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.997827    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:19:30.997992 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.997863    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:19:30.997992 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.997902    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:19:30.999697 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:30.999679    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"]
Oct 30 03:19:31.128956 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.128923    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxxpp\" (UniqueName: \"kubernetes.io/projected/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-api-access-kxxpp\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.129115 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.128964    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.129115 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.129049    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e39d25e7-de6a-4a8b-884a-236aabc50f32-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.129115 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.129078    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e39d25e7-de6a-4a8b-884a-236aabc50f32-proxy-tls\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.129251 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.129118    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.230243 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.230168    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.230243 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.230229    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxpp\" (UniqueName: \"kubernetes.io/projected/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-api-access-kxxpp\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.230421 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.230262    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.230421 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.230295    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e39d25e7-de6a-4a8b-884a-236aabc50f32-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.230421 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.230314    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e39d25e7-de6a-4a8b-884a-236aabc50f32-proxy-tls\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.230790 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.230763    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e39d25e7-de6a-4a8b-884a-236aabc50f32-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.231087 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.231061    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.231166 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.231149    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.232999 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.232972    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e39d25e7-de6a-4a8b-884a-236aabc50f32-proxy-tls\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.251489 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.251471    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxxpp\" (UniqueName: \"kubernetes.io/projected/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-api-access-kxxpp\") pod \"isvc-sklearn-predictor-5f4578d48c-qw862\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.290371 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.290350    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:31.594490 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:31.594461    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"]
Oct 30 03:19:31.596434 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:19:31.596401    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39d25e7_de6a_4a8b_884a_236aabc50f32.slice/crio-9a2ba76b2792be260d0668fbd4aa48dcf25a8291e0dd8903d663f8c57544e022 WatchSource:0}: Error finding container 9a2ba76b2792be260d0668fbd4aa48dcf25a8291e0dd8903d663f8c57544e022: Status 404 returned error can't find the container with id 9a2ba76b2792be260d0668fbd4aa48dcf25a8291e0dd8903d663f8c57544e022
Oct 30 03:19:32.110906 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.110877    4885 generic.go:356] "Generic (PLEG): container finished" podID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerID="4e577457a665e2ce44603d7a4addd386dfb1deb33553e521cbbfc4d7dfee5c1f" exitCode=137
Oct 30 03:19:32.111316 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.110951    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" event={"ID":"403a5576-f3f2-46d8-9ba0-2fb0584486ad","Type":"ContainerDied","Data":"4e577457a665e2ce44603d7a4addd386dfb1deb33553e521cbbfc4d7dfee5c1f"}
Oct 30 03:19:32.112728 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.112702    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" event={"ID":"e39d25e7-de6a-4a8b-884a-236aabc50f32","Type":"ContainerStarted","Data":"1abf089aa827b649cb42a49e6a1148c99dd632fca1076c73617a785f36e99265"}
Oct 30 03:19:32.112826 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.112762    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" event={"ID":"e39d25e7-de6a-4a8b-884a-236aabc50f32","Type":"ContainerStarted","Data":"9a2ba76b2792be260d0668fbd4aa48dcf25a8291e0dd8903d663f8c57544e022"}
Oct 30 03:19:32.213849 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.213820    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:19:32.339079 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.339002    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-api-access-jw7l6\" (UniqueName: \"kubernetes.io/projected/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-api-access-jw7l6\") pod \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") "
Oct 30 03:19:32.339079 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.339074    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-rbac-proxy-sar-config\") pod \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") "
Oct 30 03:19:32.339312 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.339178    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") "
Oct 30 03:19:32.339312 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.339219    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403a5576-f3f2-46d8-9ba0-2fb0584486ad-proxy-tls\" (UniqueName: \"kubernetes.io/secret/403a5576-f3f2-46d8-9ba0-2fb0584486ad-proxy-tls\") pod \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\" (UID: \"403a5576-f3f2-46d8-9ba0-2fb0584486ad\") "
Oct 30 03:19:32.339486 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.339462    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-rbac-proxy-sar-config" pod "403a5576-f3f2-46d8-9ba0-2fb0584486ad" (UID: "403a5576-f3f2-46d8-9ba0-2fb0584486ad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:32.341333 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.341304    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-api-access-jw7l6" pod "403a5576-f3f2-46d8-9ba0-2fb0584486ad" (UID: "403a5576-f3f2-46d8-9ba0-2fb0584486ad"). InnerVolumeSpecName "kube-api-access-jw7l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:32.341499 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.341479    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403a5576-f3f2-46d8-9ba0-2fb0584486ad-proxy-tls" pod "403a5576-f3f2-46d8-9ba0-2fb0584486ad" (UID: "403a5576-f3f2-46d8-9ba0-2fb0584486ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:32.342211 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.342191    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196" pod "403a5576-f3f2-46d8-9ba0-2fb0584486ad" (UID: "403a5576-f3f2-46d8-9ba0-2fb0584486ad"). InnerVolumeSpecName "pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:19:32.440459 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.440423    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw7l6\" (UniqueName: \"kubernetes.io/projected/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-api-access-jw7l6\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:19:32.440459 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.440453    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403a5576-f3f2-46d8-9ba0-2fb0584486ad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:19:32.440709 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:32.440477    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403a5576-f3f2-46d8-9ba0-2fb0584486ad-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:19:33.119130 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:33.119100    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"
Oct 30 03:19:33.119130 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:33.119126    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8" event={"ID":"403a5576-f3f2-46d8-9ba0-2fb0584486ad","Type":"ContainerDied","Data":"068f3919e8f202cb0069b3a418424331aa57eba8adab8293883ffe1406d89d7e"}
Oct 30 03:19:33.119710 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:33.119192    4885 scope.go:122] "RemoveContainer" containerID="ef49d10c05dff91efd775b3429ebcd2480ce812e9f7ef33b110e343caeff6243"
Oct 30 03:19:33.130040 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:33.130022    4885 scope.go:122] "RemoveContainer" containerID="4e577457a665e2ce44603d7a4addd386dfb1deb33553e521cbbfc4d7dfee5c1f"
Oct 30 03:19:33.175089 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:33.175067    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"]
Oct 30 03:19:33.178855 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:33.178836    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-5cc98f55f7-bmvl8"]
Oct 30 03:19:34.511011 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:34.510981    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"]
Oct 30 03:19:34.511474 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:34.511222    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz" podUID="463a783b-7aaf-4e2c-80d4-ab7b0883844e" containerName="kserve-container" containerID="cri-o://2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569" gracePeriod=30
Oct 30 03:19:34.623575 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:34.623548    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" path="/var/lib/kubelet/pods/403a5576-f3f2-46d8-9ba0-2fb0584486ad/volumes"
Oct 30 03:19:34.956696 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:34.956664    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"]
Oct 30 03:19:34.956959 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:34.956916    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kserve-container" containerID="cri-o://2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb" gracePeriod=30
Oct 30 03:19:34.957095 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:34.956964    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kube-rbac-proxy" containerID="cri-o://a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9" gracePeriod=30
Oct 30 03:19:35.130759 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:35.130726    4885 generic.go:356] "Generic (PLEG): container finished" podID="f61e65a3-bf78-475e-831c-65756f196c23" containerID="a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9" exitCode=0
Oct 30 03:19:35.130952 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:35.130806    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" event={"ID":"f61e65a3-bf78-475e-831c-65756f196c23","Type":"ContainerDied","Data":"a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9"}
Oct 30 03:19:36.136785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:36.136749    4885 generic.go:356] "Generic (PLEG): container finished" podID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerID="1abf089aa827b649cb42a49e6a1148c99dd632fca1076c73617a785f36e99265" exitCode=0
Oct 30 03:19:36.137187 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:36.136811    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" event={"ID":"e39d25e7-de6a-4a8b-884a-236aabc50f32","Type":"ContainerDied","Data":"1abf089aa827b649cb42a49e6a1148c99dd632fca1076c73617a785f36e99265"}
Oct 30 03:19:36.948083 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:36.948045    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Oct 30 03:19:37.144201 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:37.144168    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" event={"ID":"e39d25e7-de6a-4a8b-884a-236aabc50f32","Type":"ContainerStarted","Data":"be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9"}
Oct 30 03:19:37.144586 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:37.144228    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" event={"ID":"e39d25e7-de6a-4a8b-884a-236aabc50f32","Type":"ContainerStarted","Data":"2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de"}
Oct 30 03:19:37.144586 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:37.144416    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:37.184852 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:37.184805    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podStartSLOduration=3.234789263 podStartE2EDuration="7.184792295s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.950003032s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:37.183789741 +0000 UTC m=+1482.693958764" watchObservedRunningTime="2026-10-30 03:19:37.184792295 +0000 UTC m=+1482.694961315"
Oct 30 03:19:38.149426 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:38.149387    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:38.150377 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:38.150355    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 30 03:19:39.153564 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:39.153516    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 30 03:19:41.948259 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:41.948219    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Oct 30 03:19:44.157910 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:44.157881    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:19:44.158399 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:44.158373    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 30 03:19:46.948078 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:46.948035    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Oct 30 03:19:46.948453 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:46.948144    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:19:51.948828 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:51.948783    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Oct 30 03:19:54.158972 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:54.158929    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 30 03:19:54.762857 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:54.762827    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:19:54.765628 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:54.765589    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:19:54.768299 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:54.768279    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:19:54.770931 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:54.770911    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:19:55.921745 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:19:55.921711    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:19:56.948195 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:19:56.948153    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Oct 30 03:20:01.948899 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:01.948858    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Oct 30 03:20:04.158314 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:04.158270    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 30 03:20:04.695198 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:04.695177    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"
Oct 30 03:20:04.806731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:04.806658    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463a783b-7aaf-4e2c-80d4-ab7b0883844e-kube-api-access-8jn8d\" (UniqueName: \"kubernetes.io/projected/463a783b-7aaf-4e2c-80d4-ab7b0883844e-kube-api-access-8jn8d\") pod \"463a783b-7aaf-4e2c-80d4-ab7b0883844e\" (UID: \"463a783b-7aaf-4e2c-80d4-ab7b0883844e\") "
Oct 30 03:20:04.808804 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:04.808784    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463a783b-7aaf-4e2c-80d4-ab7b0883844e-kube-api-access-8jn8d" pod "463a783b-7aaf-4e2c-80d4-ab7b0883844e" (UID: "463a783b-7aaf-4e2c-80d4-ab7b0883844e"). InnerVolumeSpecName "kube-api-access-8jn8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:04.907583 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:04.907554    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jn8d\" (UniqueName: \"kubernetes.io/projected/463a783b-7aaf-4e2c-80d4-ab7b0883844e-kube-api-access-8jn8d\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:20:05.126135 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.126111    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:20:05.210588 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.210558    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f61e65a3-bf78-475e-831c-65756f196c23-kube-api-access-dcv7p\" (UniqueName: \"kubernetes.io/projected/f61e65a3-bf78-475e-831c-65756f196c23-kube-api-access-dcv7p\") pod \"f61e65a3-bf78-475e-831c-65756f196c23\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") "
Oct 30 03:20:05.211015 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.210602    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f61e65a3-bf78-475e-831c-65756f196c23-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f61e65a3-bf78-475e-831c-65756f196c23-proxy-tls\") pod \"f61e65a3-bf78-475e-831c-65756f196c23\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") "
Oct 30 03:20:05.211015 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.210673    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f61e65a3-bf78-475e-831c-65756f196c23-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f61e65a3-bf78-475e-831c-65756f196c23-kube-rbac-proxy-sar-config\") pod \"f61e65a3-bf78-475e-831c-65756f196c23\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") "
Oct 30 03:20:05.211015 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.210791    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") pod \"f61e65a3-bf78-475e-831c-65756f196c23\" (UID: \"f61e65a3-bf78-475e-831c-65756f196c23\") "
Oct 30 03:20:05.211177 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.211045    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f61e65a3-bf78-475e-831c-65756f196c23-kube-rbac-proxy-sar-config" pod "f61e65a3-bf78-475e-831c-65756f196c23" (UID: "f61e65a3-bf78-475e-831c-65756f196c23"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:05.213030 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.213002    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61e65a3-bf78-475e-831c-65756f196c23-kube-api-access-dcv7p" pod "f61e65a3-bf78-475e-831c-65756f196c23" (UID: "f61e65a3-bf78-475e-831c-65756f196c23"). InnerVolumeSpecName "kube-api-access-dcv7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:05.213151 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.213005    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f61e65a3-bf78-475e-831c-65756f196c23-proxy-tls" pod "f61e65a3-bf78-475e-831c-65756f196c23" (UID: "f61e65a3-bf78-475e-831c-65756f196c23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:05.213756 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.213737    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196" pod "f61e65a3-bf78-475e-831c-65756f196c23" (UID: "f61e65a3-bf78-475e-831c-65756f196c23"). InnerVolumeSpecName "pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:20:05.275042 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.275014    4885 generic.go:356] "Generic (PLEG): container finished" podID="f61e65a3-bf78-475e-831c-65756f196c23" containerID="2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb" exitCode=137
Oct 30 03:20:05.275200 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.275063    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"
Oct 30 03:20:05.275200 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.275097    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" event={"ID":"f61e65a3-bf78-475e-831c-65756f196c23","Type":"ContainerDied","Data":"2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb"}
Oct 30 03:20:05.275200 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.275159    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn" event={"ID":"f61e65a3-bf78-475e-831c-65756f196c23","Type":"ContainerDied","Data":"8c81aeacad2eb2c18e0e751dd7cc2ef2b87f747ef2103d77d1eaa0a5d1b42298"}
Oct 30 03:20:05.275200 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.275188    4885 scope.go:122] "RemoveContainer" containerID="a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9"
Oct 30 03:20:05.276417 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.276392    4885 generic.go:356] "Generic (PLEG): container finished" podID="463a783b-7aaf-4e2c-80d4-ab7b0883844e" containerID="2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569" exitCode=137
Oct 30 03:20:05.276500 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.276427    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz" event={"ID":"463a783b-7aaf-4e2c-80d4-ab7b0883844e","Type":"ContainerDied","Data":"2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569"}
Oct 30 03:20:05.276500 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.276441    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"
Oct 30 03:20:05.276500 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.276457    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz" event={"ID":"463a783b-7aaf-4e2c-80d4-ab7b0883844e","Type":"ContainerDied","Data":"605e4cff2caf18b107e304536d4bef70547919d843bf8361f3b83cbc360a240c"}
Oct 30 03:20:05.283785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.283736    4885 scope.go:122] "RemoveContainer" containerID="2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb"
Oct 30 03:20:05.292045 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.292028    4885 scope.go:122] "RemoveContainer" containerID="a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9"
Oct 30 03:20:05.292306 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:20:05.292288    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9\": container with ID starting with a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9 not found: ID does not exist" containerID="a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9"
Oct 30 03:20:05.292349 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.292314    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9"} err="failed to get container status \"a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9\": rpc error: code = NotFound desc = could not find container \"a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9\": container with ID starting with a39beb69f145ca7a2caedcfbbe893c6d0eabbda26ae4f50f99ca0c9c1fb89bf9 not found: ID does not exist"
Oct 30 03:20:05.292349 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.292331    4885 scope.go:122] "RemoveContainer" containerID="2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb"
Oct 30 03:20:05.292566 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:20:05.292550    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb\": container with ID starting with 2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb not found: ID does not exist" containerID="2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb"
Oct 30 03:20:05.292604 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.292575    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb"} err="failed to get container status \"2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb\": rpc error: code = NotFound desc = could not find container \"2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb\": container with ID starting with 2171c3e07aacdab60e896cfca3aa2c63a677a316dfd8d68ccb490df712581ceb not found: ID does not exist"
Oct 30 03:20:05.292604 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.292589    4885 scope.go:122] "RemoveContainer" containerID="2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569"
Oct 30 03:20:05.299837 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.299821    4885 scope.go:122] "RemoveContainer" containerID="2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569"
Oct 30 03:20:05.300118 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:20:05.300091    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569\": container with ID starting with 2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569 not found: ID does not exist" containerID="2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569"
Oct 30 03:20:05.300162 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.300125    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569"} err="failed to get container status \"2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569\": rpc error: code = NotFound desc = could not find container \"2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569\": container with ID starting with 2dacc999072d142b826c1306fbda9cb3bd6d196a45e42be5b0d8da05e728a569 not found: ID does not exist"
Oct 30 03:20:05.312390 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.312341    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcv7p\" (UniqueName: \"kubernetes.io/projected/f61e65a3-bf78-475e-831c-65756f196c23-kube-api-access-dcv7p\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:20:05.312390 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.312361    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f61e65a3-bf78-475e-831c-65756f196c23-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:20:05.312390 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.312371    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f61e65a3-bf78-475e-831c-65756f196c23-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:20:05.312535 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.312398    4885 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") on node \"ip-10-0-140-132.ec2.internal\" "
Oct 30 03:20:05.320439 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.320415    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"]
Oct 30 03:20:05.320607 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.320595    4885 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196") on node "ip-10-0-140-132.ec2.internal"
Oct 30 03:20:05.324044 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.324024    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-predictor-678db9d655-g62rn"]
Oct 30 03:20:05.352961 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.352938    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"]
Oct 30 03:20:05.358269 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.358250    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-21889b45-sink-745bfdcbfd-qz9pz"]
Oct 30 03:20:05.413590 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:05.413558    4885 reconciler_common.go:299] "Volume detached for volume \"pvc-22ba4385-aadb-4e1d-96c6-1a5f90a56883\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aee23e6efa1c1196\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"csi-90cd0c3d0c6e9d2561aa4eeb9e4df42f8a45df9c3822250dcd8b46f166be07ec\""
Oct 30 03:20:06.468539 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.468506    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"]
Oct 30 03:20:06.468930 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.468914    4885 cpu_manager.go:414] "containerMap: removing container" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kube-rbac-proxy"
Oct 30 03:20:06.468968 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.468954    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kube-rbac-proxy"
Oct 30 03:20:06.469021 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.468983    4885 cpu_manager.go:414] "containerMap: removing container" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kserve-container"
Oct 30 03:20:06.469021 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.468991    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kserve-container"
Oct 30 03:20:06.469021 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.469005    4885 cpu_manager.go:414] "containerMap: removing container" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kserve-container"
Oct 30 03:20:06.469021 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.469012    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kserve-container"
Oct 30 03:20:06.469021 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.469022    4885 cpu_manager.go:414] "containerMap: removing container" podUID="463a783b-7aaf-4e2c-80d4-ab7b0883844e" containerName="kserve-container"
Oct 30 03:20:06.469188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.469027    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463a783b-7aaf-4e2c-80d4-ab7b0883844e" containerName="kserve-container"
Oct 30 03:20:06.469188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.469047    4885 cpu_manager.go:414] "containerMap: removing container" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kube-rbac-proxy"
Oct 30 03:20:06.469188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.469052    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kube-rbac-proxy"
Oct 30 03:20:06.469188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.469113    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kube-rbac-proxy"
Oct 30 03:20:06.469188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.469123    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="f61e65a3-bf78-475e-831c-65756f196c23" containerName="kserve-container"
Oct 30 03:20:06.469188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.469130    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kserve-container"
Oct 30 03:20:06.469188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.469136    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="463a783b-7aaf-4e2c-80d4-ab7b0883844e" containerName="kserve-container"
Oct 30 03:20:06.469188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.469145    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="403a5576-f3f2-46d8-9ba0-2fb0584486ad" containerName="kube-rbac-proxy"
Oct 30 03:20:06.473761 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.473738    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"
Oct 30 03:20:06.497752 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.497727    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:20:06.497901 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.497776    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:20:06.497901 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.497777    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:20:06.499108 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.499091    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"]
Oct 30 03:20:06.524944 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.524914    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7xgb\" (UniqueName: \"kubernetes.io/projected/a6d26769-ea7c-4bf9-bff6-08ec73de8441-kube-api-access-n7xgb\") pod \"proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5\" (UID: \"a6d26769-ea7c-4bf9-bff6-08ec73de8441\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"
Oct 30 03:20:06.623965 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.623928    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="463a783b-7aaf-4e2c-80d4-ab7b0883844e" path="/var/lib/kubelet/pods/463a783b-7aaf-4e2c-80d4-ab7b0883844e/volumes"
Oct 30 03:20:06.624394 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.624374    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f61e65a3-bf78-475e-831c-65756f196c23" path="/var/lib/kubelet/pods/f61e65a3-bf78-475e-831c-65756f196c23/volumes"
Oct 30 03:20:06.626260 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.626236    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xgb\" (UniqueName: \"kubernetes.io/projected/a6d26769-ea7c-4bf9-bff6-08ec73de8441-kube-api-access-n7xgb\") pod \"proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5\" (UID: \"a6d26769-ea7c-4bf9-bff6-08ec73de8441\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"
Oct 30 03:20:06.648238 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.648213    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7xgb\" (UniqueName: \"kubernetes.io/projected/a6d26769-ea7c-4bf9-bff6-08ec73de8441-kube-api-access-n7xgb\") pod \"proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5\" (UID: \"a6d26769-ea7c-4bf9-bff6-08ec73de8441\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"
Oct 30 03:20:06.785409 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:06.785335    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"
Oct 30 03:20:07.076435 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:07.076321    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"]
Oct 30 03:20:07.078605 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:20:07.078577    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d26769_ea7c_4bf9_bff6_08ec73de8441.slice/crio-d9eaa5431bbf9a82345d9c4ca1f341cf7d3c648caec0d889dae40d537681ace2 WatchSource:0}: Error finding container d9eaa5431bbf9a82345d9c4ca1f341cf7d3c648caec0d889dae40d537681ace2: Status 404 returned error can't find the container with id d9eaa5431bbf9a82345d9c4ca1f341cf7d3c648caec0d889dae40d537681ace2
Oct 30 03:20:07.288797 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:07.288765    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5" event={"ID":"a6d26769-ea7c-4bf9-bff6-08ec73de8441","Type":"ContainerStarted","Data":"6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e"}
Oct 30 03:20:07.288978 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:07.288821    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5" event={"ID":"a6d26769-ea7c-4bf9-bff6-08ec73de8441","Type":"ContainerStarted","Data":"d9eaa5431bbf9a82345d9c4ca1f341cf7d3c648caec0d889dae40d537681ace2"}
Oct 30 03:20:07.288978 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:07.288921    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"
Oct 30 03:20:07.290637 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:07.290598    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5" podUID="a6d26769-ea7c-4bf9-bff6-08ec73de8441" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.61:8080/\": dial tcp 10.132.0.61:8080: connect: connection refused"
Oct 30 03:20:07.326919 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:07.326814    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5" podStartSLOduration=1.3267949749999999 podStartE2EDuration="1.326794975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:07.325588205 +0000 UTC m=+1512.835757250" watchObservedRunningTime="2026-10-30 03:20:07.326794975 +0000 UTC m=+1512.836964004"
Oct 30 03:20:08.294606 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:08.294577    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"
Oct 30 03:20:12.186251 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.186219    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"]
Oct 30 03:20:12.190330 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.190313    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:12.203276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.203247    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-3aaa824d-kube-rbac-proxy-sar-config\""
Oct 30 03:20:12.213209 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.213179    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-3aaa824d-dockercfg-fwxcs\""
Oct 30 03:20:12.213325 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.213261    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"proxy-routing-3aaa824d-predictor-serving-cert\""
Oct 30 03:20:12.214838 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.214817    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"]
Oct 30 03:20:12.271934 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.271892    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4befab22-4889-43af-b9ea-5633a011c248-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:12.272061 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.272013    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4befab22-4889-43af-b9ea-5633a011c248-proxy-tls\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:12.272111 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.272061    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vj58\" (UniqueName: \"kubernetes.io/projected/4befab22-4889-43af-b9ea-5633a011c248-kube-api-access-4vj58\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:12.373203 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.373172    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4befab22-4889-43af-b9ea-5633a011c248-proxy-tls\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:12.373343 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.373211    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vj58\" (UniqueName: \"kubernetes.io/projected/4befab22-4889-43af-b9ea-5633a011c248-kube-api-access-4vj58\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:12.373343 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.373251    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4befab22-4889-43af-b9ea-5633a011c248-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:12.373895 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.373874    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4befab22-4889-43af-b9ea-5633a011c248-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:12.375741 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.375721    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4befab22-4889-43af-b9ea-5633a011c248-proxy-tls\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:12.397174 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:12.397146    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vj58\" (UniqueName: \"kubernetes.io/projected/4befab22-4889-43af-b9ea-5633a011c248-kube-api-access-4vj58\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:14.159178 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:14.159144    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 30 03:20:17.713642 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:17.713599    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:17.738401 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:17.738375    4885 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") device path: \"\"" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:17.814332 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:17.814307    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:17.814475 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:17.814457    4885 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") DevicePath \"\"" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:17.829245 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:17.829223    4885 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") DevicePath \"csi-836c149a508d2ab4903bf1b8a95cc6139bb8df6aa348cf76012deab8d2d1cb4c\"" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:18.006262 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:18.006234    4885 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/30e37bd05bc568dd375a5f41788bf21afcf254eb3d3ef42086701256f4bb849f/globalmount\"" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:18.028791 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:18.028762    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-848577b8bc-9rthk\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:18.206633 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:18.206564    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:18.512568 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:18.512540    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"]
Oct 30 03:20:18.515318 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:20:18.515293    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4befab22_4889_43af_b9ea_5633a011c248.slice/crio-c189d4a9675bb887f857ec45e9024973b570ba0d304ed9caaacc143833417b7f WatchSource:0}: Error finding container c189d4a9675bb887f857ec45e9024973b570ba0d304ed9caaacc143833417b7f: Status 404 returned error can't find the container with id c189d4a9675bb887f857ec45e9024973b570ba0d304ed9caaacc143833417b7f
Oct 30 03:20:19.342880 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:19.342850    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" event={"ID":"4befab22-4889-43af-b9ea-5633a011c248","Type":"ContainerStarted","Data":"4ec525dc7740440e6772600f64e6fcac75ba3604cf11b156fcc8251c0554d686"}
Oct 30 03:20:19.343321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:19.342910    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" event={"ID":"4befab22-4889-43af-b9ea-5633a011c248","Type":"ContainerStarted","Data":"42e18b05e26033bf5915a742ee38e2aabc0e44b1d08196614616ff807626c805"}
Oct 30 03:20:19.343321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:19.342949    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:19.343321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:19.343086    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:19.343321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:19.343142    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:19.343321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:19.343177    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" event={"ID":"4befab22-4889-43af-b9ea-5633a011c248","Type":"ContainerStarted","Data":"80e8da30fbe3f6aa6e4b8c9a5f80d2d11a7f985abeac3483787046aff3efd87f"}
Oct 30 03:20:19.343321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:19.343201    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" event={"ID":"4befab22-4889-43af-b9ea-5633a011c248","Type":"ContainerStarted","Data":"c189d4a9675bb887f857ec45e9024973b570ba0d304ed9caaacc143833417b7f"}
Oct 30 03:20:19.345356 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:19.345336    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:19.345797 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:19.345783    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:19.437586 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:19.437544    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podStartSLOduration=10.437531554 podStartE2EDuration="10.437531554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:19.396479373 +0000 UTC m=+1524.906648404" watchObservedRunningTime="2026-10-30 03:20:19.437531554 +0000 UTC m=+1524.947700580"
Oct 30 03:20:24.159286 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:24.159252    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 30 03:20:25.922506 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:20:25.922465    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:20:26.356171 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:26.356144    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:28.992687 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:28.992657    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"]
Oct 30 03:20:28.997511 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:28.997492    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.021847 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.021808    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"]
Oct 30 03:20:29.108203 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.108176    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe719be-d42b-4390-90b2-f650cbcdaf86-proxy-tls\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.108347 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.108211    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.108347 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.108308    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.108426 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.108346    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvck5\" (UniqueName: \"kubernetes.io/projected/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-api-access-qvck5\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.108472 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.108452    4885 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") DevicePath \"csi-836c149a508d2ab4903bf1b8a95cc6139bb8df6aa348cf76012deab8d2d1cb4c\"" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.120572 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.120552    4885 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") DevicePath \"csi-836c149a508d2ab4903bf1b8a95cc6139bb8df6aa348cf76012deab8d2d1cb4c\"" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.141449 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.141425    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.209486 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.209450    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvck5\" (UniqueName: \"kubernetes.io/projected/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-api-access-qvck5\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.209715 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.209505    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe719be-d42b-4390-90b2-f650cbcdaf86-proxy-tls\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.209715 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.209591    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.210275 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.210256    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.212072 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.212055    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe719be-d42b-4390-90b2-f650cbcdaf86-proxy-tls\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.230935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.230914    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvck5\" (UniqueName: \"kubernetes.io/projected/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-api-access-qvck5\") pod \"proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.314066 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.313994    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:29.619711 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:29.619682    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"]
Oct 30 03:20:29.621984 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:20:29.621957    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe719be_d42b_4390_90b2_f650cbcdaf86.slice/crio-057795148ede231024c069479ad4c6c0484ac80425daaf16f837105b1f981e9a WatchSource:0}: Error finding container 057795148ede231024c069479ad4c6c0484ac80425daaf16f837105b1f981e9a: Status 404 returned error can't find the container with id 057795148ede231024c069479ad4c6c0484ac80425daaf16f837105b1f981e9a
Oct 30 03:20:30.393661 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:30.393630    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" event={"ID":"5fe719be-d42b-4390-90b2-f650cbcdaf86","Type":"ContainerStarted","Data":"3d46d437c871958df268d787f37e2702f576706be574760efd01fd30eba780fe"}
Oct 30 03:20:30.394030 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:30.393686    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" event={"ID":"5fe719be-d42b-4390-90b2-f650cbcdaf86","Type":"ContainerStarted","Data":"c5bcf5fed322c809861f8284618104c6310085215bd39f09aa2fac8bbca836bd"}
Oct 30 03:20:30.394030 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:30.393717    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" event={"ID":"5fe719be-d42b-4390-90b2-f650cbcdaf86","Type":"ContainerStarted","Data":"731d6a21b4ca50ae89781da58a6d5cdf7cdc898b1d2a313beb64a231ba511df6"}
Oct 30 03:20:30.394030 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:30.393739    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" event={"ID":"5fe719be-d42b-4390-90b2-f650cbcdaf86","Type":"ContainerStarted","Data":"057795148ede231024c069479ad4c6c0484ac80425daaf16f837105b1f981e9a"}
Oct 30 03:20:30.394030 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:30.393796    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:30.454943 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:30.454896    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podStartSLOduration=2.454881785 podStartE2EDuration="2.454881785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:30.444106119 +0000 UTC m=+1535.954275156" watchObservedRunningTime="2026-10-30 03:20:30.454881785 +0000 UTC m=+1535.965050813"
Oct 30 03:20:31.398342 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:31.398305    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:31.398815 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:31.398372    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:31.400800 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:31.400776    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:31.401271 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:31.401255    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:34.158305 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:34.158260    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 30 03:20:38.413869 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:38.413843    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:20:38.490544 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:38.490514    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"]
Oct 30 03:20:38.490838 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:38.490805    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kserve-container" containerID="cri-o://80e8da30fbe3f6aa6e4b8c9a5f80d2d11a7f985abeac3483787046aff3efd87f" gracePeriod=30
Oct 30 03:20:38.491030 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:38.490835    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="agent" containerID="cri-o://4ec525dc7740440e6772600f64e6fcac75ba3604cf11b156fcc8251c0554d686" gracePeriod=30
Oct 30 03:20:38.491030 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:38.490977    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kube-rbac-proxy" containerID="cri-o://42e18b05e26033bf5915a742ee38e2aabc0e44b1d08196614616ff807626c805" gracePeriod=30
Oct 30 03:20:39.344444 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:39.344407    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:20:39.439664 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:39.439607    4885 generic.go:356] "Generic (PLEG): container finished" podID="4befab22-4889-43af-b9ea-5633a011c248" containerID="42e18b05e26033bf5915a742ee38e2aabc0e44b1d08196614616ff807626c805" exitCode=0
Oct 30 03:20:39.439664 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:39.439651    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" event={"ID":"4befab22-4889-43af-b9ea-5633a011c248","Type":"ContainerDied","Data":"42e18b05e26033bf5915a742ee38e2aabc0e44b1d08196614616ff807626c805"}
Oct 30 03:20:41.352850 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:41.352811    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Oct 30 03:20:44.158446 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:44.158405    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 30 03:20:46.353051 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:46.353010    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Oct 30 03:20:49.343702 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:49.343662    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:20:51.352311 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:51.352266    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Oct 30 03:20:51.352761 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:51.352394    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:20:54.159164 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:54.159136    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:20:55.921859 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:20:55.921823    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:20:56.352857 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:56.352816    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Oct 30 03:20:59.344121 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:59.344082    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:20:59.344513 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:20:59.344210    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:21:01.302359 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:01.302332    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:01.352636 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:01.352584    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Oct 30 03:21:03.346415 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:03.346390    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:05.380870 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:05.380838    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:06.352850 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:06.352766    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Oct 30 03:21:07.414492 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:07.414463    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:08.571708 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.571673    4885 generic.go:356] "Generic (PLEG): container finished" podID="4befab22-4889-43af-b9ea-5633a011c248" containerID="4ec525dc7740440e6772600f64e6fcac75ba3604cf11b156fcc8251c0554d686" exitCode=137
Oct 30 03:21:08.571708 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.571703    4885 generic.go:356] "Generic (PLEG): container finished" podID="4befab22-4889-43af-b9ea-5633a011c248" containerID="80e8da30fbe3f6aa6e4b8c9a5f80d2d11a7f985abeac3483787046aff3efd87f" exitCode=137
Oct 30 03:21:08.572109 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.571726    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" event={"ID":"4befab22-4889-43af-b9ea-5633a011c248","Type":"ContainerDied","Data":"4ec525dc7740440e6772600f64e6fcac75ba3604cf11b156fcc8251c0554d686"}
Oct 30 03:21:08.572109 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.571786    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" event={"ID":"4befab22-4889-43af-b9ea-5633a011c248","Type":"ContainerDied","Data":"80e8da30fbe3f6aa6e4b8c9a5f80d2d11a7f985abeac3483787046aff3efd87f"}
Oct 30 03:21:08.661385 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.661360    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:21:08.771452 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.771377    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4befab22-4889-43af-b9ea-5633a011c248-kube-api-access-4vj58\" (UniqueName: \"kubernetes.io/projected/4befab22-4889-43af-b9ea-5633a011c248-kube-api-access-4vj58\") pod \"4befab22-4889-43af-b9ea-5633a011c248\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") "
Oct 30 03:21:08.771452 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.771443    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4befab22-4889-43af-b9ea-5633a011c248-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4befab22-4889-43af-b9ea-5633a011c248-kube-rbac-proxy-sar-config\") pod \"4befab22-4889-43af-b9ea-5633a011c248\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") "
Oct 30 03:21:08.771676 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.771477    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4befab22-4889-43af-b9ea-5633a011c248-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4befab22-4889-43af-b9ea-5633a011c248-proxy-tls\") pod \"4befab22-4889-43af-b9ea-5633a011c248\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") "
Oct 30 03:21:08.771676 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.771554    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"4befab22-4889-43af-b9ea-5633a011c248\" (UID: \"4befab22-4889-43af-b9ea-5633a011c248\") "
Oct 30 03:21:08.771794 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.771766    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4befab22-4889-43af-b9ea-5633a011c248-kube-rbac-proxy-sar-config" pod "4befab22-4889-43af-b9ea-5633a011c248" (UID: "4befab22-4889-43af-b9ea-5633a011c248"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:08.774314 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.774293    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4befab22-4889-43af-b9ea-5633a011c248-proxy-tls" pod "4befab22-4889-43af-b9ea-5633a011c248" (UID: "4befab22-4889-43af-b9ea-5633a011c248"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:08.774432 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.774330    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4befab22-4889-43af-b9ea-5633a011c248-kube-api-access-4vj58" pod "4befab22-4889-43af-b9ea-5633a011c248" (UID: "4befab22-4889-43af-b9ea-5633a011c248"). InnerVolumeSpecName "kube-api-access-4vj58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:08.774432 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.774339    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b" pod "4befab22-4889-43af-b9ea-5633a011c248" (UID: "4befab22-4889-43af-b9ea-5633a011c248"). InnerVolumeSpecName "pvc-557cea0b-f581-497f-83f6-1416691dd2ce". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:21:08.873092 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.873068    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4befab22-4889-43af-b9ea-5633a011c248-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:21:08.873092 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.873090    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4befab22-4889-43af-b9ea-5633a011c248-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:21:08.873248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:08.873099    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vj58\" (UniqueName: \"kubernetes.io/projected/4befab22-4889-43af-b9ea-5633a011c248-kube-api-access-4vj58\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:21:09.447760 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:09.447731    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:09.578564 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:09.578540    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"
Oct 30 03:21:09.579070 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:09.578539    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk" event={"ID":"4befab22-4889-43af-b9ea-5633a011c248","Type":"ContainerDied","Data":"c189d4a9675bb887f857ec45e9024973b570ba0d304ed9caaacc143833417b7f"}
Oct 30 03:21:09.579070 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:09.578752    4885 scope.go:122] "RemoveContainer" containerID="4ec525dc7740440e6772600f64e6fcac75ba3604cf11b156fcc8251c0554d686"
Oct 30 03:21:09.588973 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:09.588949    4885 scope.go:122] "RemoveContainer" containerID="42e18b05e26033bf5915a742ee38e2aabc0e44b1d08196614616ff807626c805"
Oct 30 03:21:09.601203 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:09.601174    4885 scope.go:122] "RemoveContainer" containerID="80e8da30fbe3f6aa6e4b8c9a5f80d2d11a7f985abeac3483787046aff3efd87f"
Oct 30 03:21:09.636174 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:09.636154    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"]
Oct 30 03:21:09.646309 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:09.646288    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-848577b8bc-9rthk"]
Oct 30 03:21:10.624432 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.624399    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4befab22-4889-43af-b9ea-5633a011c248" path="/var/lib/kubelet/pods/4befab22-4889-43af-b9ea-5633a011c248/volumes"
Oct 30 03:21:10.953572 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.953500    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"]
Oct 30 03:21:10.953977 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.953960    4885 cpu_manager.go:414] "containerMap: removing container" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kserve-container"
Oct 30 03:21:10.954025 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.953979    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kserve-container"
Oct 30 03:21:10.954025 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.953997    4885 cpu_manager.go:414] "containerMap: removing container" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="agent"
Oct 30 03:21:10.954025 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.954002    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="agent"
Oct 30 03:21:10.954025 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.954019    4885 cpu_manager.go:414] "containerMap: removing container" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kube-rbac-proxy"
Oct 30 03:21:10.954025 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.954023    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kube-rbac-proxy"
Oct 30 03:21:10.954192 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.954082    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="agent"
Oct 30 03:21:10.954192 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.954092    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kube-rbac-proxy"
Oct 30 03:21:10.954192 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.954100    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4befab22-4889-43af-b9ea-5633a011c248" containerName="kserve-container"
Oct 30 03:21:10.958666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.958646    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:10.985360 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:10.985324    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"]
Oct 30 03:21:11.091677 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.091643    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kbcw\" (UniqueName: \"kubernetes.io/projected/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-api-access-8kbcw\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.091857 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.091691    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.091857 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.091759    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f84c74-806f-4c87-987c-bb0e774ff9b8-proxy-tls\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.091857 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.091775    4885 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") DevicePath \"csi-836c149a508d2ab4903bf1b8a95cc6139bb8df6aa348cf76012deab8d2d1cb4c\"" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.091857 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.091786    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.107234 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.107203    4885 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") DevicePath \"csi-836c149a508d2ab4903bf1b8a95cc6139bb8df6aa348cf76012deab8d2d1cb4c\"" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.132264 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.132230    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.192668 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.192641    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f84c74-806f-4c87-987c-bb0e774ff9b8-proxy-tls\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.192810 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.192674    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.192810 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.192741    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kbcw\" (UniqueName: \"kubernetes.io/projected/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-api-access-8kbcw\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.193402 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.193329    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.195249 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.195224    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f84c74-806f-4c87-987c-bb0e774ff9b8-proxy-tls\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.215424 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.215397    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kbcw\" (UniqueName: \"kubernetes.io/projected/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-api-access-8kbcw\") pod \"proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") " pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.274926 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.274908    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:11.480815 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.480740    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:11.586662 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:11.586640    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"]
Oct 30 03:21:11.588802 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:21:11.588776    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-c5e0cc2bf0cee77160f970d719ed5aea84c5f38bedf9051df628447ff8d36184 WatchSource:0}: Error finding container c5e0cc2bf0cee77160f970d719ed5aea84c5f38bedf9051df628447ff8d36184: Status 404 returned error can't find the container with id c5e0cc2bf0cee77160f970d719ed5aea84c5f38bedf9051df628447ff8d36184
Oct 30 03:21:12.595509 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:12.595471    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" event={"ID":"22f84c74-806f-4c87-987c-bb0e774ff9b8","Type":"ContainerStarted","Data":"3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350"}
Oct 30 03:21:12.596208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:12.595535    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" event={"ID":"22f84c74-806f-4c87-987c-bb0e774ff9b8","Type":"ContainerStarted","Data":"bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e"}
Oct 30 03:21:12.596208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:12.595707    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:12.596208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:12.595778    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:12.596208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:12.595814    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" event={"ID":"22f84c74-806f-4c87-987c-bb0e774ff9b8","Type":"ContainerStarted","Data":"c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971"}
Oct 30 03:21:12.596208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:12.595852    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" event={"ID":"22f84c74-806f-4c87-987c-bb0e774ff9b8","Type":"ContainerStarted","Data":"c5e0cc2bf0cee77160f970d719ed5aea84c5f38bedf9051df628447ff8d36184"}
Oct 30 03:21:12.596208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:12.595880    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:12.598112 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:12.598093    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:12.598554 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:12.598541    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:12.664315 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:12.664266    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podStartSLOduration=2.66425152 podStartE2EDuration="2.66425152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:12.653250419 +0000 UTC m=+1578.163419456" watchObservedRunningTime="2026-10-30 03:21:12.66425152 +0000 UTC m=+1578.174420545"
Oct 30 03:21:13.528882 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:13.528849    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:15.564455 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:15.564423    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:17.597984 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:17.597952    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:19.609731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:19.609700    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:21:19.641935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:19.641909    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:19.695887 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:19.695853    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"]
Oct 30 03:21:19.696182 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:19.696155    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kserve-container" containerID="cri-o://731d6a21b4ca50ae89781da58a6d5cdf7cdc898b1d2a313beb64a231ba511df6" gracePeriod=30
Oct 30 03:21:19.696293 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:19.696273    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kube-rbac-proxy" containerID="cri-o://c5bcf5fed322c809861f8284618104c6310085215bd39f09aa2fac8bbca836bd" gracePeriod=30
Oct 30 03:21:19.696397 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:19.696365    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="agent" containerID="cri-o://3d46d437c871958df268d787f37e2702f576706be574760efd01fd30eba780fe" gracePeriod=30
Oct 30 03:21:20.633379 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:20.633346    4885 generic.go:356] "Generic (PLEG): container finished" podID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerID="c5bcf5fed322c809861f8284618104c6310085215bd39f09aa2fac8bbca836bd" exitCode=0
Oct 30 03:21:20.633786 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:20.633425    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" event={"ID":"5fe719be-d42b-4390-90b2-f650cbcdaf86","Type":"ContainerDied","Data":"c5bcf5fed322c809861f8284618104c6310085215bd39f09aa2fac8bbca836bd"}
Oct 30 03:21:21.398933 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:21.398897    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:21:21.675513 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:21.675434    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:23.409793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:23.409758    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Oct 30 03:21:23.709063 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:23.709037    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:25.744995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:25.744964    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:25.921599 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:21:25.921566    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:21:27.782270 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:27.782212    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:28.409778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:28.409731    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Oct 30 03:21:29.825355 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:29.825327    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:31.398724 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:31.398686    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:21:31.858420 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:31.858389    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-predictor-5f4578d48c-qw862_e39d25e7-de6a-4a8b-884a-236aabc50f32/kserve-container/0.log"
Oct 30 03:21:32.413507 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:32.413461    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"]
Oct 30 03:21:32.413926 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:32.413801    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container" containerID="cri-o://2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de" gracePeriod=30
Oct 30 03:21:32.414008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:32.413913    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kube-rbac-proxy" containerID="cri-o://be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9" gracePeriod=30
Oct 30 03:21:32.690000 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:32.689913    4885 generic.go:356] "Generic (PLEG): container finished" podID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerID="be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9" exitCode=0
Oct 30 03:21:32.690000 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:32.689970    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" event={"ID":"e39d25e7-de6a-4a8b-884a-236aabc50f32","Type":"ContainerDied","Data":"be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9"}
Oct 30 03:21:33.409750 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:33.409697    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Oct 30 03:21:33.409926 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:33.409858    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:21:34.154384 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:34.154342    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Oct 30 03:21:34.158605 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:34.158580    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 30 03:21:37.073561 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.073537    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:21:37.120959 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.120880    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e39d25e7-de6a-4a8b-884a-236aabc50f32-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e39d25e7-de6a-4a8b-884a-236aabc50f32-proxy-tls\") pod \"e39d25e7-de6a-4a8b-884a-236aabc50f32\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") "
Oct 30 03:21:37.120959 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.120929    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39d25e7-de6a-4a8b-884a-236aabc50f32-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e39d25e7-de6a-4a8b-884a-236aabc50f32-kserve-provision-location\") pod \"e39d25e7-de6a-4a8b-884a-236aabc50f32\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") "
Oct 30 03:21:37.121167 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.120963    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-rbac-proxy-sar-config\") pod \"e39d25e7-de6a-4a8b-884a-236aabc50f32\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") "
Oct 30 03:21:37.121167 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.121038    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-api-access-kxxpp\" (UniqueName: \"kubernetes.io/projected/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-api-access-kxxpp\") pod \"e39d25e7-de6a-4a8b-884a-236aabc50f32\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") "
Oct 30 03:21:37.121167 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.121061    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-cabundle-cert\") pod \"e39d25e7-de6a-4a8b-884a-236aabc50f32\" (UID: \"e39d25e7-de6a-4a8b-884a-236aabc50f32\") "
Oct 30 03:21:37.121334 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.121275    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39d25e7-de6a-4a8b-884a-236aabc50f32-kserve-provision-location" pod "e39d25e7-de6a-4a8b-884a-236aabc50f32" (UID: "e39d25e7-de6a-4a8b-884a-236aabc50f32"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:37.121441 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.121413    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-rbac-proxy-sar-config" pod "e39d25e7-de6a-4a8b-884a-236aabc50f32" (UID: "e39d25e7-de6a-4a8b-884a-236aabc50f32"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:37.121748 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.121467    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-cabundle-cert" pod "e39d25e7-de6a-4a8b-884a-236aabc50f32" (UID: "e39d25e7-de6a-4a8b-884a-236aabc50f32"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:37.123177 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.123141    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e39d25e7-de6a-4a8b-884a-236aabc50f32-proxy-tls" pod "e39d25e7-de6a-4a8b-884a-236aabc50f32" (UID: "e39d25e7-de6a-4a8b-884a-236aabc50f32"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:37.123254 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.123229    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-api-access-kxxpp" pod "e39d25e7-de6a-4a8b-884a-236aabc50f32" (UID: "e39d25e7-de6a-4a8b-884a-236aabc50f32"). InnerVolumeSpecName "kube-api-access-kxxpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:37.222571 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.222538    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxxpp\" (UniqueName: \"kubernetes.io/projected/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-api-access-kxxpp\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:21:37.222571 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.222570    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:21:37.222799 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.222583    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e39d25e7-de6a-4a8b-884a-236aabc50f32-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:21:37.222799 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.222594    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e39d25e7-de6a-4a8b-884a-236aabc50f32-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:21:37.222799 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.222606    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e39d25e7-de6a-4a8b-884a-236aabc50f32-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:21:37.715450 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.715419    4885 generic.go:356] "Generic (PLEG): container finished" podID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerID="2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de" exitCode=0
Oct 30 03:21:37.715653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.715496    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"
Oct 30 03:21:37.715653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.715509    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" event={"ID":"e39d25e7-de6a-4a8b-884a-236aabc50f32","Type":"ContainerDied","Data":"2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de"}
Oct 30 03:21:37.715653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.715565    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862" event={"ID":"e39d25e7-de6a-4a8b-884a-236aabc50f32","Type":"ContainerDied","Data":"9a2ba76b2792be260d0668fbd4aa48dcf25a8291e0dd8903d663f8c57544e022"}
Oct 30 03:21:37.715653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.715597    4885 scope.go:122] "RemoveContainer" containerID="be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9"
Oct 30 03:21:37.724790 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.724772    4885 scope.go:122] "RemoveContainer" containerID="2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de"
Oct 30 03:21:37.738736 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.738714    4885 scope.go:122] "RemoveContainer" containerID="1abf089aa827b649cb42a49e6a1148c99dd632fca1076c73617a785f36e99265"
Oct 30 03:21:37.746477 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.746460    4885 scope.go:122] "RemoveContainer" containerID="be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9"
Oct 30 03:21:37.746778 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:21:37.746754    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9\": container with ID starting with be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9 not found: ID does not exist" containerID="be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9"
Oct 30 03:21:37.746879 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.746783    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9"} err="failed to get container status \"be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9\": rpc error: code = NotFound desc = could not find container \"be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9\": container with ID starting with be3ef2a483a800cbbf4ddb0537fd253e9e4ed8b09a752fe556c07b8420ea11c9 not found: ID does not exist"
Oct 30 03:21:37.746879 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.746801    4885 scope.go:122] "RemoveContainer" containerID="2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de"
Oct 30 03:21:37.747058 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:21:37.747040    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de\": container with ID starting with 2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de not found: ID does not exist" containerID="2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de"
Oct 30 03:21:37.747100 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.747065    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de"} err="failed to get container status \"2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de\": rpc error: code = NotFound desc = could not find container \"2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de\": container with ID starting with 2335a5336757389b0c3033e8f9f7d0057e5686420087a7285cada51067f062de not found: ID does not exist"
Oct 30 03:21:37.747100 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.747078    4885 scope.go:122] "RemoveContainer" containerID="1abf089aa827b649cb42a49e6a1148c99dd632fca1076c73617a785f36e99265"
Oct 30 03:21:37.747305 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:21:37.747289    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1abf089aa827b649cb42a49e6a1148c99dd632fca1076c73617a785f36e99265\": container with ID starting with 1abf089aa827b649cb42a49e6a1148c99dd632fca1076c73617a785f36e99265 not found: ID does not exist" containerID="1abf089aa827b649cb42a49e6a1148c99dd632fca1076c73617a785f36e99265"
Oct 30 03:21:37.747344 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.747312    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abf089aa827b649cb42a49e6a1148c99dd632fca1076c73617a785f36e99265"} err="failed to get container status \"1abf089aa827b649cb42a49e6a1148c99dd632fca1076c73617a785f36e99265\": rpc error: code = NotFound desc = could not find container \"1abf089aa827b649cb42a49e6a1148c99dd632fca1076c73617a785f36e99265\": container with ID starting with 1abf089aa827b649cb42a49e6a1148c99dd632fca1076c73617a785f36e99265 not found: ID does not exist"
Oct 30 03:21:37.772921 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.772890    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"]
Oct 30 03:21:37.782964 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:37.782942    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-predictor-5f4578d48c-qw862"]
Oct 30 03:21:38.410110 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:38.410070    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Oct 30 03:21:38.624464 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:38.624424    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" path="/var/lib/kubelet/pods/e39d25e7-de6a-4a8b-884a-236aabc50f32/volumes"
Oct 30 03:21:40.123513 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.123480    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"]
Oct 30 03:21:40.124040 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.123857    4885 cpu_manager.go:414] "containerMap: removing container" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="storage-initializer"
Oct 30 03:21:40.124040 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.123869    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="storage-initializer"
Oct 30 03:21:40.124040 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.123885    4885 cpu_manager.go:414] "containerMap: removing container" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container"
Oct 30 03:21:40.124040 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.123890    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container"
Oct 30 03:21:40.124040 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.123929    4885 cpu_manager.go:414] "containerMap: removing container" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kube-rbac-proxy"
Oct 30 03:21:40.124040 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.123936    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kube-rbac-proxy"
Oct 30 03:21:40.124040 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.124001    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kserve-container"
Oct 30 03:21:40.124040 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.124015    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39d25e7-de6a-4a8b-884a-236aabc50f32" containerName="kube-rbac-proxy"
Oct 30 03:21:40.127541 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.127522    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.144711 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.144683    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgvgc\" (UniqueName: \"kubernetes.io/projected/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-api-access-rgvgc\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.144841 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.144717    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.144841 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.144738    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.144841 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.144764    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.145012 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.144921    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.150628 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.150585    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:21:40.150832 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.150587    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:21:40.150832 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.150697    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:21:40.150832 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.150701    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Oct 30 03:21:40.150832 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.150796    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:21:40.150832 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.150823    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 03:21:40.151021 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.150860    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:21:40.152805 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.152782    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"]
Oct 30 03:21:40.245607 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.245578    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgvgc\" (UniqueName: \"kubernetes.io/projected/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-api-access-rgvgc\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.245807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.245634    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.245807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.245666    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.245807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.245685    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.245807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.245718    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.246134 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.246115    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.246422 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.246400    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.246472 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.246439    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.248263 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.248247    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.267198 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.267170    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgvgc\" (UniqueName: \"kubernetes.io/projected/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-api-access-rgvgc\") pod \"sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") " pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.445133 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.445055    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:21:40.752933 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:40.752913    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"]
Oct 30 03:21:40.754932 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:21:40.754906    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387e4eef_2f7c_463c_9ffd_4f3a09adfe7a.slice/crio-bb83166465c8b0c01d09e9136dd5ff97ce1afd9f95b359a890757b5c6b700e73 WatchSource:0}: Error finding container bb83166465c8b0c01d09e9136dd5ff97ce1afd9f95b359a890757b5c6b700e73: Status 404 returned error can't find the container with id bb83166465c8b0c01d09e9136dd5ff97ce1afd9f95b359a890757b5c6b700e73
Oct 30 03:21:41.399221 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:41.399185    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:21:41.399710 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:41.399322    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:21:41.735849 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:41.735807    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" event={"ID":"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a","Type":"ContainerStarted","Data":"61b061c2b758bfc474de6ffddecb299fb4d6fc90288139f24d99dc00dc92d9b4"}
Oct 30 03:21:41.736250 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:41.735879    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" event={"ID":"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a","Type":"ContainerStarted","Data":"bb83166465c8b0c01d09e9136dd5ff97ce1afd9f95b359a890757b5c6b700e73"}
Oct 30 03:21:43.409810 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:43.409767    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Oct 30 03:21:44.751539 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:44.751506    4885 generic.go:356] "Generic (PLEG): container finished" podID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerID="61b061c2b758bfc474de6ffddecb299fb4d6fc90288139f24d99dc00dc92d9b4" exitCode=0
Oct 30 03:21:44.751946 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:44.751577    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" event={"ID":"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a","Type":"ContainerDied","Data":"61b061c2b758bfc474de6ffddecb299fb4d6fc90288139f24d99dc00dc92d9b4"}
Oct 30 03:21:44.752753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:44.752738    4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:21:48.410338 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:48.410279    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Oct 30 03:21:49.800910 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:49.800875    4885 generic.go:356] "Generic (PLEG): container finished" podID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerID="3d46d437c871958df268d787f37e2702f576706be574760efd01fd30eba780fe" exitCode=137
Oct 30 03:21:49.800910 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:49.800906    4885 generic.go:356] "Generic (PLEG): container finished" podID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerID="731d6a21b4ca50ae89781da58a6d5cdf7cdc898b1d2a313beb64a231ba511df6" exitCode=137
Oct 30 03:21:49.801436 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:49.801006    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" event={"ID":"5fe719be-d42b-4390-90b2-f650cbcdaf86","Type":"ContainerDied","Data":"3d46d437c871958df268d787f37e2702f576706be574760efd01fd30eba780fe"}
Oct 30 03:21:49.801436 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:49.801075    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" event={"ID":"5fe719be-d42b-4390-90b2-f650cbcdaf86","Type":"ContainerDied","Data":"731d6a21b4ca50ae89781da58a6d5cdf7cdc898b1d2a313beb64a231ba511df6"}
Oct 30 03:21:49.950966 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:49.950900    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:21:50.033402 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.033367    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-api-access-qvck5\" (UniqueName: \"kubernetes.io/projected/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-api-access-qvck5\") pod \"5fe719be-d42b-4390-90b2-f650cbcdaf86\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") "
Oct 30 03:21:50.033600 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.033495    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"5fe719be-d42b-4390-90b2-f650cbcdaf86\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") "
Oct 30 03:21:50.033600 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.033582    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fe719be-d42b-4390-90b2-f650cbcdaf86-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe719be-d42b-4390-90b2-f650cbcdaf86-proxy-tls\") pod \"5fe719be-d42b-4390-90b2-f650cbcdaf86\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") "
Oct 30 03:21:50.033748 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.033675    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-rbac-proxy-sar-config\") pod \"5fe719be-d42b-4390-90b2-f650cbcdaf86\" (UID: \"5fe719be-d42b-4390-90b2-f650cbcdaf86\") "
Oct 30 03:21:50.034312 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.034278    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-rbac-proxy-sar-config" pod "5fe719be-d42b-4390-90b2-f650cbcdaf86" (UID: "5fe719be-d42b-4390-90b2-f650cbcdaf86"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:50.041937 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.041905    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-api-access-qvck5" pod "5fe719be-d42b-4390-90b2-f650cbcdaf86" (UID: "5fe719be-d42b-4390-90b2-f650cbcdaf86"). InnerVolumeSpecName "kube-api-access-qvck5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:50.042377 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.042350    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe719be-d42b-4390-90b2-f650cbcdaf86-proxy-tls" pod "5fe719be-d42b-4390-90b2-f650cbcdaf86" (UID: "5fe719be-d42b-4390-90b2-f650cbcdaf86"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:50.042560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.042524    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b" pod "5fe719be-d42b-4390-90b2-f650cbcdaf86" (UID: "5fe719be-d42b-4390-90b2-f650cbcdaf86"). InnerVolumeSpecName "pvc-557cea0b-f581-497f-83f6-1416691dd2ce". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:21:50.134520 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.134470    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe719be-d42b-4390-90b2-f650cbcdaf86-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:21:50.134520 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.134502    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:21:50.134520 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.134514    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvck5\" (UniqueName: \"kubernetes.io/projected/5fe719be-d42b-4390-90b2-f650cbcdaf86-kube-api-access-qvck5\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:21:50.811865 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.811822    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp" event={"ID":"5fe719be-d42b-4390-90b2-f650cbcdaf86","Type":"ContainerDied","Data":"057795148ede231024c069479ad4c6c0484ac80425daaf16f837105b1f981e9a"}
Oct 30 03:21:50.812316 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.811909    4885 scope.go:122] "RemoveContainer" containerID="3d46d437c871958df268d787f37e2702f576706be574760efd01fd30eba780fe"
Oct 30 03:21:50.812316 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.812099    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"
Oct 30 03:21:50.824840 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.824815    4885 scope.go:122] "RemoveContainer" containerID="c5bcf5fed322c809861f8284618104c6310085215bd39f09aa2fac8bbca836bd"
Oct 30 03:21:50.835818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.835801    4885 scope.go:122] "RemoveContainer" containerID="731d6a21b4ca50ae89781da58a6d5cdf7cdc898b1d2a313beb64a231ba511df6"
Oct 30 03:21:50.879967 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.879936    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"]
Oct 30 03:21:50.888899 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:50.888842    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-86d9b9f845-s9fjp"]
Oct 30 03:21:51.505386 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:51.505355    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"]
Oct 30 03:21:51.505581 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:51.505560    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5" podUID="a6d26769-ea7c-4bf9-bff6-08ec73de8441" containerName="kserve-container" containerID="cri-o://6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e" gracePeriod=30
Oct 30 03:21:51.934485 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:51.934413    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"]
Oct 30 03:21:51.935003 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:51.934901    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kserve-container" containerID="cri-o://c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971" gracePeriod=30
Oct 30 03:21:51.935176 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:51.935143    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="agent" containerID="cri-o://3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350" gracePeriod=30
Oct 30 03:21:51.935330 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:51.935244    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kube-rbac-proxy" containerID="cri-o://bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e" gracePeriod=30
Oct 30 03:21:52.597419 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:52.597376    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:21:52.627548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:52.627513    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" path="/var/lib/kubelet/pods/5fe719be-d42b-4390-90b2-f650cbcdaf86/volumes"
Oct 30 03:21:52.828158 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:52.827978    4885 generic.go:356] "Generic (PLEG): container finished" podID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerID="bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e" exitCode=0
Oct 30 03:21:52.828158 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:52.828095    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" event={"ID":"22f84c74-806f-4c87-987c-bb0e774ff9b8","Type":"ContainerDied","Data":"bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e"}
Oct 30 03:21:54.605967 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:54.605922    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Oct 30 03:21:55.934148 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:21:55.934104    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:21:59.605844 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:21:59.605804    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Oct 30 03:22:02.597175 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:02.597119    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:22:04.606100 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:04.606056    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Oct 30 03:22:04.606526 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:04.606174    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:22:09.605918 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:09.605873    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Oct 30 03:22:12.597007 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:12.596957    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:22:12.597416 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:12.597081    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:22:14.605674 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:14.605633    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Oct 30 03:22:19.605328 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:19.605283    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Oct 30 03:22:21.741626 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:21.741582    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"
Oct 30 03:22:21.817653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:21.817554    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d26769-ea7c-4bf9-bff6-08ec73de8441-kube-api-access-n7xgb\" (UniqueName: \"kubernetes.io/projected/a6d26769-ea7c-4bf9-bff6-08ec73de8441-kube-api-access-n7xgb\") pod \"a6d26769-ea7c-4bf9-bff6-08ec73de8441\" (UID: \"a6d26769-ea7c-4bf9-bff6-08ec73de8441\") "
Oct 30 03:22:21.820005 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:21.819977    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d26769-ea7c-4bf9-bff6-08ec73de8441-kube-api-access-n7xgb" pod "a6d26769-ea7c-4bf9-bff6-08ec73de8441" (UID: "a6d26769-ea7c-4bf9-bff6-08ec73de8441"). InnerVolumeSpecName "kube-api-access-n7xgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:21.918787 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:21.918750    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7xgb\" (UniqueName: \"kubernetes.io/projected/a6d26769-ea7c-4bf9-bff6-08ec73de8441-kube-api-access-n7xgb\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:22:22.008934 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:22:22.008874    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d26769_ea7c_4bf9_bff6_08ec73de8441.slice/crio-6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-conmon-c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d26769_ea7c_4bf9_bff6_08ec73de8441.slice/crio-conmon-6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-conmon-3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:22:22.008934 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:22:22.008908    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d26769_ea7c_4bf9_bff6_08ec73de8441.slice/crio-6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-conmon-c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-conmon-3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d26769_ea7c_4bf9_bff6_08ec73de8441.slice/crio-conmon-6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:22:22.009190 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:22:22.008990    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-conmon-3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-conmon-c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d26769_ea7c_4bf9_bff6_08ec73de8441.slice/crio-conmon-6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:22:22.009190 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:22:22.008990    4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-c5e0cc2bf0cee77160f970d719ed5aea84c5f38bedf9051df628447ff8d36184\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d26769_ea7c_4bf9_bff6_08ec73de8441.slice/crio-6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d26769_ea7c_4bf9_bff6_08ec73de8441.slice/crio-conmon-6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-conmon-c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f84c74_806f_4c87_987c_bb0e774ff9b8.slice/crio-conmon-3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:22:22.014092 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.014064    4885 generic.go:356] "Generic (PLEG): container finished" podID="a6d26769-ea7c-4bf9-bff6-08ec73de8441" containerID="6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e" exitCode=137
Oct 30 03:22:22.014270 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.014105    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5" event={"ID":"a6d26769-ea7c-4bf9-bff6-08ec73de8441","Type":"ContainerDied","Data":"6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e"}
Oct 30 03:22:22.014270 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.014124    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"
Oct 30 03:22:22.014270 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.014182    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5" event={"ID":"a6d26769-ea7c-4bf9-bff6-08ec73de8441","Type":"ContainerDied","Data":"d9eaa5431bbf9a82345d9c4ca1f341cf7d3c648caec0d889dae40d537681ace2"}
Oct 30 03:22:22.014270 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.014208    4885 scope.go:122] "RemoveContainer" containerID="6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e"
Oct 30 03:22:22.028465 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.028447    4885 scope.go:122] "RemoveContainer" containerID="6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e"
Oct 30 03:22:22.029073 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:22:22.029047    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e\": container with ID starting with 6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e not found: ID does not exist" containerID="6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e"
Oct 30 03:22:22.029187 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.029084    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e"} err="failed to get container status \"6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e\": rpc error: code = NotFound desc = could not find container \"6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e\": container with ID starting with 6ef8ad11fbfea6a2cb8a5389f4813dfac07598b77d8ec8c6816f4b2c856e684e not found: ID does not exist"
Oct 30 03:22:22.060567 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.060530    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"]
Oct 30 03:22:22.064226 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.064193    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-sink-5c947d94f7-fmgx5"]
Oct 30 03:22:22.175956 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.175929    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:22:22.321091 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.321057    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-api-access-8kbcw\" (UniqueName: \"kubernetes.io/projected/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-api-access-8kbcw\") pod \"22f84c74-806f-4c87-987c-bb0e774ff9b8\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") "
Oct 30 03:22:22.321257 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.321195    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") pod \"22f84c74-806f-4c87-987c-bb0e774ff9b8\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") "
Oct 30 03:22:22.321300 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.321261    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-rbac-proxy-sar-config\") pod \"22f84c74-806f-4c87-987c-bb0e774ff9b8\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") "
Oct 30 03:22:22.321335 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.321302    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22f84c74-806f-4c87-987c-bb0e774ff9b8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f84c74-806f-4c87-987c-bb0e774ff9b8-proxy-tls\") pod \"22f84c74-806f-4c87-987c-bb0e774ff9b8\" (UID: \"22f84c74-806f-4c87-987c-bb0e774ff9b8\") "
Oct 30 03:22:22.321639 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.321586    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-rbac-proxy-sar-config" pod "22f84c74-806f-4c87-987c-bb0e774ff9b8" (UID: "22f84c74-806f-4c87-987c-bb0e774ff9b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:22.323420 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.323397    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-api-access-8kbcw" pod "22f84c74-806f-4c87-987c-bb0e774ff9b8" (UID: "22f84c74-806f-4c87-987c-bb0e774ff9b8"). InnerVolumeSpecName "kube-api-access-8kbcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:22.323542 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.323464    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f84c74-806f-4c87-987c-bb0e774ff9b8-proxy-tls" pod "22f84c74-806f-4c87-987c-bb0e774ff9b8" (UID: "22f84c74-806f-4c87-987c-bb0e774ff9b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:22:22.324054 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.324034    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b" pod "22f84c74-806f-4c87-987c-bb0e774ff9b8" (UID: "22f84c74-806f-4c87-987c-bb0e774ff9b8"). InnerVolumeSpecName "pvc-557cea0b-f581-497f-83f6-1416691dd2ce". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:22:22.422806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.422720    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kbcw\" (UniqueName: \"kubernetes.io/projected/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-api-access-8kbcw\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:22:22.422806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.422781    4885 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") on node \"ip-10-0-140-132.ec2.internal\" "
Oct 30 03:22:22.422806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.422797    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f84c74-806f-4c87-987c-bb0e774ff9b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:22:22.423098 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.422812    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f84c74-806f-4c87-987c-bb0e774ff9b8-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:22:22.434135 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.434110    4885 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-557cea0b-f581-497f-83f6-1416691dd2ce" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b") on node "ip-10-0-140-132.ec2.internal"
Oct 30 03:22:22.523212 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.523181    4885 reconciler_common.go:299] "Volume detached for volume \"pvc-557cea0b-f581-497f-83f6-1416691dd2ce\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c0a679b23e1c96b\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"csi-836c149a508d2ab4903bf1b8a95cc6139bb8df6aa348cf76012deab8d2d1cb4c\""
Oct 30 03:22:22.625715 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:22.625684    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d26769-ea7c-4bf9-bff6-08ec73de8441" path="/var/lib/kubelet/pods/a6d26769-ea7c-4bf9-bff6-08ec73de8441/volumes"
Oct 30 03:22:23.023908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.023863    4885 generic.go:356] "Generic (PLEG): container finished" podID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerID="3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350" exitCode=0
Oct 30 03:22:23.023908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.023906    4885 generic.go:356] "Generic (PLEG): container finished" podID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerID="c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971" exitCode=137
Oct 30 03:22:23.024449 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.023940    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" event={"ID":"22f84c74-806f-4c87-987c-bb0e774ff9b8","Type":"ContainerDied","Data":"3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350"}
Oct 30 03:22:23.024449 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.024005    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" event={"ID":"22f84c74-806f-4c87-987c-bb0e774ff9b8","Type":"ContainerDied","Data":"c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971"}
Oct 30 03:22:23.024449 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.023951    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"
Oct 30 03:22:23.024449 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.024043    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6" event={"ID":"22f84c74-806f-4c87-987c-bb0e774ff9b8","Type":"ContainerDied","Data":"c5e0cc2bf0cee77160f970d719ed5aea84c5f38bedf9051df628447ff8d36184"}
Oct 30 03:22:23.024449 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.024083    4885 scope.go:122] "RemoveContainer" containerID="3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350"
Oct 30 03:22:23.035336 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.035317    4885 scope.go:122] "RemoveContainer" containerID="bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e"
Oct 30 03:22:23.044409 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.044393    4885 scope.go:122] "RemoveContainer" containerID="c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971"
Oct 30 03:22:23.054238 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.054208    4885 scope.go:122] "RemoveContainer" containerID="3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350"
Oct 30 03:22:23.054549 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:22:23.054522    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350\": container with ID starting with 3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350 not found: ID does not exist" containerID="3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350"
Oct 30 03:22:23.054724 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.054561    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350"} err="failed to get container status \"3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350\": rpc error: code = NotFound desc = could not find container \"3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350\": container with ID starting with 3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350 not found: ID does not exist"
Oct 30 03:22:23.054724 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.054586    4885 scope.go:122] "RemoveContainer" containerID="bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e"
Oct 30 03:22:23.054921 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:22:23.054899    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e\": container with ID starting with bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e not found: ID does not exist" containerID="bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e"
Oct 30 03:22:23.055023 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.054928    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e"} err="failed to get container status \"bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e\": rpc error: code = NotFound desc = could not find container \"bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e\": container with ID starting with bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e not found: ID does not exist"
Oct 30 03:22:23.055023 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.054944    4885 scope.go:122] "RemoveContainer" containerID="c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971"
Oct 30 03:22:23.055220 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:22:23.055201    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971\": container with ID starting with c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971 not found: ID does not exist" containerID="c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971"
Oct 30 03:22:23.055269 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.055229    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971"} err="failed to get container status \"c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971\": rpc error: code = NotFound desc = could not find container \"c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971\": container with ID starting with c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971 not found: ID does not exist"
Oct 30 03:22:23.055269 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.055246    4885 scope.go:122] "RemoveContainer" containerID="3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350"
Oct 30 03:22:23.055563 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.055536    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350"} err="failed to get container status \"3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350\": rpc error: code = NotFound desc = could not find container \"3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350\": container with ID starting with 3a96b175985b6bd64d67fdc4b46f6fac4430949fae45e9efd4d6142f720b8350 not found: ID does not exist"
Oct 30 03:22:23.055650 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.055563    4885 scope.go:122] "RemoveContainer" containerID="bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e"
Oct 30 03:22:23.055837 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.055818    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e"} err="failed to get container status \"bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e\": rpc error: code = NotFound desc = could not find container \"bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e\": container with ID starting with bb6f5b42cc7ea1539204e0960a6412247faccc96a7879d2812963b1416ce364e not found: ID does not exist"
Oct 30 03:22:23.055837 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.055836    4885 scope.go:122] "RemoveContainer" containerID="c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971"
Oct 30 03:22:23.056144 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.056127    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971"} err="failed to get container status \"c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971\": rpc error: code = NotFound desc = could not find container \"c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971\": container with ID starting with c0c3bd362314559fe55b77ee6a1af83635a37819218085c070446913ab30e971 not found: ID does not exist"
Oct 30 03:22:23.081367 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.081042    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"]
Oct 30 03:22:23.091255 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:23.091231    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/proxy-routing-3aaa824d-predictor-75dddf6d9d-t58c6"]
Oct 30 03:22:24.240999 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.240969    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"]
Oct 30 03:22:24.241504 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241483    4885 cpu_manager.go:414] "containerMap: removing container" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kserve-container"
Oct 30 03:22:24.241596 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241508    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kserve-container"
Oct 30 03:22:24.241596 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241527    4885 cpu_manager.go:414] "containerMap: removing container" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kserve-container"
Oct 30 03:22:24.241596 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241535    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kserve-container"
Oct 30 03:22:24.241596 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241557    4885 cpu_manager.go:414] "containerMap: removing container" podUID="a6d26769-ea7c-4bf9-bff6-08ec73de8441" containerName="kserve-container"
Oct 30 03:22:24.241596 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241565    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d26769-ea7c-4bf9-bff6-08ec73de8441" containerName="kserve-container"
Oct 30 03:22:24.241596 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241584    4885 cpu_manager.go:414] "containerMap: removing container" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kube-rbac-proxy"
Oct 30 03:22:24.241596 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241592    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kube-rbac-proxy"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241628    4885 cpu_manager.go:414] "containerMap: removing container" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="agent"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241636    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="agent"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241653    4885 cpu_manager.go:414] "containerMap: removing container" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="agent"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241661    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="agent"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241674    4885 cpu_manager.go:414] "containerMap: removing container" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kube-rbac-proxy"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241680    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kube-rbac-proxy"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241773    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d26769-ea7c-4bf9-bff6-08ec73de8441" containerName="kserve-container"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241786    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kube-rbac-proxy"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241798    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="agent"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241809    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="agent"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241818    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kube-rbac-proxy"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241829    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" containerName="kserve-container"
Oct 30 03:22:24.241970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.241839    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe719be-d42b-4390-90b2-f650cbcdaf86" containerName="kserve-container"
Oct 30 03:22:24.277743 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.277710    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"]
Oct 30 03:22:24.277923 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.277878    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:24.300903 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.300876    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:22:24.300903 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.300888    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:22:24.301097 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.300888    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:22:24.301097 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.300888    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Oct 30 03:22:24.301097 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.300972    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"message-dumper-predictor-serving-cert\""
Oct 30 03:22:24.437911 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.437871    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12fcc0c8-59a1-4cf7-8445-1ef34984f036-proxy-tls\") pod \"message-dumper-predictor-5d746b9b7c-752th\" (UID: \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\") " pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:24.438080 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.438007    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-5d746b9b7c-752th\" (UID: \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\") " pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:24.438080 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.438045    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6k7p\" (UniqueName: \"kubernetes.io/projected/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-api-access-p6k7p\") pod \"message-dumper-predictor-5d746b9b7c-752th\" (UID: \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\") " pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:24.539438 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.539353    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12fcc0c8-59a1-4cf7-8445-1ef34984f036-proxy-tls\") pod \"message-dumper-predictor-5d746b9b7c-752th\" (UID: \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\") " pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:24.539438 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.539424    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-5d746b9b7c-752th\" (UID: \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\") " pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:24.539719 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.539468    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6k7p\" (UniqueName: \"kubernetes.io/projected/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-api-access-p6k7p\") pod \"message-dumper-predictor-5d746b9b7c-752th\" (UID: \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\") " pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:24.540316 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.540226    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-5d746b9b7c-752th\" (UID: \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\") " pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:24.542213 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.542190    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12fcc0c8-59a1-4cf7-8445-1ef34984f036-proxy-tls\") pod \"message-dumper-predictor-5d746b9b7c-752th\" (UID: \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\") " pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:24.561684 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.561652    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6k7p\" (UniqueName: \"kubernetes.io/projected/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-api-access-p6k7p\") pod \"message-dumper-predictor-5d746b9b7c-752th\" (UID: \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\") " pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:24.625827 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.625791    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f84c74-806f-4c87-987c-bb0e774ff9b8" path="/var/lib/kubelet/pods/22f84c74-806f-4c87-987c-bb0e774ff9b8/volumes"
Oct 30 03:22:24.631040 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.631020    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:24.954586 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:24.954535    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"]
Oct 30 03:22:24.957873 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:22:24.957845    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12fcc0c8_59a1_4cf7_8445_1ef34984f036.slice/crio-09fe8d64856503aaec84d5a426e3ee6176b62fda043f66b8876de01e42b72dbf WatchSource:0}: Error finding container 09fe8d64856503aaec84d5a426e3ee6176b62fda043f66b8876de01e42b72dbf: Status 404 returned error can't find the container with id 09fe8d64856503aaec84d5a426e3ee6176b62fda043f66b8876de01e42b72dbf
Oct 30 03:22:25.037819 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:25.037778    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th" event={"ID":"12fcc0c8-59a1-4cf7-8445-1ef34984f036","Type":"ContainerStarted","Data":"09fe8d64856503aaec84d5a426e3ee6176b62fda043f66b8876de01e42b72dbf"}
Oct 30 03:22:25.929427 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:22:25.929386    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:22:27.051110 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:27.051073    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th" event={"ID":"12fcc0c8-59a1-4cf7-8445-1ef34984f036","Type":"ContainerStarted","Data":"e06ff6a27d86dc7626a30dc6b5e5f2a23536dd7cd9545a151c091f5864989d1a"}
Oct 30 03:22:28.056907 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:28.056876    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th" event={"ID":"12fcc0c8-59a1-4cf7-8445-1ef34984f036","Type":"ContainerStarted","Data":"58dc8ba1e2abb07331a524b496335ea03fc7af5332ed7e0132f3dae16b6f207c"}
Oct 30 03:22:28.057343 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:28.057018    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:28.095153 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:28.095086    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th" podStartSLOduration=2.135680436 podStartE2EDuration="4.09507074s" totalImagesPullingTime="1.959390304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:28.094963868 +0000 UTC m=+1653.605132925" watchObservedRunningTime="2026-10-30 03:22:28.09507074 +0000 UTC m=+1653.605239767"
Oct 30 03:22:29.062948 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:29.062918    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:29.065254 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:29.064996    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:36.079223 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:36.079130    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:22:44.487712 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.487679    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"]
Oct 30 03:22:44.490855 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.490835    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.504932 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.504880    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:22:44.504932 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.504881    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Oct 30 03:22:44.505123 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.504949    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-logger-predictor-serving-cert\""
Oct 30 03:22:44.515148 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.515111    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:22:44.515349 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.515324    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-cabundle-cert\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.515480 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.515399    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/139693d0-5632-4330-a851-dd512cc6a707-kserve-provision-location\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.515480 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.515429    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.515480 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.515457    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4886z\" (UniqueName: \"kubernetes.io/projected/139693d0-5632-4330-a851-dd512cc6a707-kube-api-access-4886z\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.515649 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.515510    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/139693d0-5632-4330-a851-dd512cc6a707-proxy-tls\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.517528 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.517508    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"]
Oct 30 03:22:44.615964 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.615933    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/139693d0-5632-4330-a851-dd512cc6a707-kserve-provision-location\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.615964 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.615972    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.616199 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.615993    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4886z\" (UniqueName: \"kubernetes.io/projected/139693d0-5632-4330-a851-dd512cc6a707-kube-api-access-4886z\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.616199 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.616114    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/139693d0-5632-4330-a851-dd512cc6a707-proxy-tls\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.616199 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.616181    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-cabundle-cert\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.616386 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.616371    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/139693d0-5632-4330-a851-dd512cc6a707-kserve-provision-location\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.616731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.616714    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-cabundle-cert\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.616794 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.616753    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.618967 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.618946    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/139693d0-5632-4330-a851-dd512cc6a707-proxy-tls\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.638842 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.638814    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4886z\" (UniqueName: \"kubernetes.io/projected/139693d0-5632-4330-a851-dd512cc6a707-kube-api-access-4886z\") pod \"isvc-logger-predictor-78555b5bb7-6j55z\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") " pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:44.809380 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:44.809298    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:45.127313 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:45.127231    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"]
Oct 30 03:22:45.130790 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:22:45.130754    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139693d0_5632_4330_a851_dd512cc6a707.slice/crio-19279a49a37a658b5a876e19c28288a31ec04ebc25cc251bd690359a6181fb19 WatchSource:0}: Error finding container 19279a49a37a658b5a876e19c28288a31ec04ebc25cc251bd690359a6181fb19: Status 404 returned error can't find the container with id 19279a49a37a658b5a876e19c28288a31ec04ebc25cc251bd690359a6181fb19
Oct 30 03:22:45.147668 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:45.147600    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" event={"ID":"139693d0-5632-4330-a851-dd512cc6a707","Type":"ContainerStarted","Data":"19279a49a37a658b5a876e19c28288a31ec04ebc25cc251bd690359a6181fb19"}
Oct 30 03:22:46.155238 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:46.155206    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" event={"ID":"139693d0-5632-4330-a851-dd512cc6a707","Type":"ContainerStarted","Data":"701f4c1de9c9ff4fa409598d37aff4d08058bf6f1c0522536b1ac2ae389be362"}
Oct 30 03:22:50.177572 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:50.177531    4885 generic.go:356] "Generic (PLEG): container finished" podID="139693d0-5632-4330-a851-dd512cc6a707" containerID="701f4c1de9c9ff4fa409598d37aff4d08058bf6f1c0522536b1ac2ae389be362" exitCode=0
Oct 30 03:22:50.178123 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:50.177632    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" event={"ID":"139693d0-5632-4330-a851-dd512cc6a707","Type":"ContainerDied","Data":"701f4c1de9c9ff4fa409598d37aff4d08058bf6f1c0522536b1ac2ae389be362"}
Oct 30 03:22:51.186530 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:51.186491    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" event={"ID":"139693d0-5632-4330-a851-dd512cc6a707","Type":"ContainerStarted","Data":"30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef"}
Oct 30 03:22:51.187018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:51.186571    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" event={"ID":"139693d0-5632-4330-a851-dd512cc6a707","Type":"ContainerStarted","Data":"051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4"}
Oct 30 03:22:51.187018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:51.186604    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" event={"ID":"139693d0-5632-4330-a851-dd512cc6a707","Type":"ContainerStarted","Data":"03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba"}
Oct 30 03:22:51.187018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:51.186955    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:51.187018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:51.187016    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:51.188884 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:51.188848    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:22:51.249117 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:51.249063    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podStartSLOduration=3.274167199 podStartE2EDuration="7.249045648s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.974878449s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:51.23873691 +0000 UTC m=+1676.748905956" watchObservedRunningTime="2026-10-30 03:22:51.249045648 +0000 UTC m=+1676.759214672"
Oct 30 03:22:52.191359 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:52.191320    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:52.191832 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:52.191495    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:22:52.192593 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:52.192564    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:22:53.196887 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:53.196839    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:22:53.197435 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:53.197409    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:22:55.925517 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:22:55.925472    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:22:58.202484 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:58.202457    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:22:58.203174 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:58.203135    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:22:58.203528 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:22:58.203500    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:23:08.203972 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:08.203923    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:23:08.204417 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:08.204359    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:23:18.203803 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:18.203756    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:23:18.204463 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:18.204293    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:23:25.929446 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:23:25.929411    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:23:28.203984 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:28.203943    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:23:28.204503 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:28.204473    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:23:38.203960 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:38.203915    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:23:38.204552 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:38.204458    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:23:48.203298 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:48.203169    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:23:48.204063 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:48.203607    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:23:51.499535 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:51.499497    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" event={"ID":"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a","Type":"ContainerStarted","Data":"adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d"}
Oct 30 03:23:51.500081 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:51.499555    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" event={"ID":"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a","Type":"ContainerStarted","Data":"b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5"}
Oct 30 03:23:51.500081 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:51.499661    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:23:51.551400 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:51.551347    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" podStartSLOduration=1.561292138 podStartE2EDuration="2m11.551332196s" totalImagesPullingTime="2m6.287330274s" totalInitContainerRuntime="3.702709784s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:51.550008089 +0000 UTC m=+1737.060177139" watchObservedRunningTime="2026-10-30 03:23:51.551332196 +0000 UTC m=+1737.061501289"
Oct 30 03:23:52.504661 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:52.504632    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:23:55.921452 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:23:55.921414    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:23:58.203496 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:58.203457    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:23:58.204027 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:58.204004    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:23:58.515138 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:23:58.515109    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:24:02.619448 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:02.619401    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:24:02.619921 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:02.619799    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:24:12.620206 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:12.620121    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:24:12.620739 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:12.620591    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:24:22.624255 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:22.624229    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:24:22.624929 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:22.624319    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:24:25.923072 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:24:25.923030    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:24:28.534573 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:28.534536    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 03:24:29.788169 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:29.788134    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-4a54dc1685_message-dumper-predictor-5d746b9b7c-752th_12fcc0c8-59a1-4cf7-8445-1ef34984f036/kserve-container/0.log"
Oct 30 03:24:30.422505 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.422454    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"]
Oct 30 03:24:30.422866 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.422833    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" containerID="cri-o://03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba" gracePeriod=30
Oct 30 03:24:30.423267 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.423237    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" containerID="cri-o://30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef" gracePeriod=30
Oct 30 03:24:30.423378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.423359    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kube-rbac-proxy" containerID="cri-o://051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4" gracePeriod=30
Oct 30 03:24:30.523427 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.523391    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"]
Oct 30 03:24:30.523729 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.523683    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th" podUID="12fcc0c8-59a1-4cf7-8445-1ef34984f036" containerName="kserve-container" containerID="cri-o://e06ff6a27d86dc7626a30dc6b5e5f2a23536dd7cd9545a151c091f5864989d1a" gracePeriod=30
Oct 30 03:24:30.523826 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.523777    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th" podUID="12fcc0c8-59a1-4cf7-8445-1ef34984f036" containerName="kube-rbac-proxy" containerID="cri-o://58dc8ba1e2abb07331a524b496335ea03fc7af5332ed7e0132f3dae16b6f207c" gracePeriod=30
Oct 30 03:24:30.675358 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.675273    4885 generic.go:356] "Generic (PLEG): container finished" podID="12fcc0c8-59a1-4cf7-8445-1ef34984f036" containerID="58dc8ba1e2abb07331a524b496335ea03fc7af5332ed7e0132f3dae16b6f207c" exitCode=0
Oct 30 03:24:30.675358 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.675300    4885 generic.go:356] "Generic (PLEG): container finished" podID="12fcc0c8-59a1-4cf7-8445-1ef34984f036" containerID="e06ff6a27d86dc7626a30dc6b5e5f2a23536dd7cd9545a151c091f5864989d1a" exitCode=2
Oct 30 03:24:30.675358 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.675344    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th" event={"ID":"12fcc0c8-59a1-4cf7-8445-1ef34984f036","Type":"ContainerDied","Data":"58dc8ba1e2abb07331a524b496335ea03fc7af5332ed7e0132f3dae16b6f207c"}
Oct 30 03:24:30.675597 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.675414    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th" event={"ID":"12fcc0c8-59a1-4cf7-8445-1ef34984f036","Type":"ContainerDied","Data":"e06ff6a27d86dc7626a30dc6b5e5f2a23536dd7cd9545a151c091f5864989d1a"}
Oct 30 03:24:30.677912 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.677882    4885 generic.go:356] "Generic (PLEG): container finished" podID="139693d0-5632-4330-a851-dd512cc6a707" containerID="051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4" exitCode=0
Oct 30 03:24:30.678047 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.677912    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" event={"ID":"139693d0-5632-4330-a851-dd512cc6a707","Type":"ContainerDied","Data":"051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4"}
Oct 30 03:24:30.769328 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.769306    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:24:30.935151 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.935064    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-rbac-proxy-sar-config\") pod \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\" (UID: \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\") "
Oct 30 03:24:30.935151 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.935122    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-api-access-p6k7p\" (UniqueName: \"kubernetes.io/projected/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-api-access-p6k7p\") pod \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\" (UID: \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\") "
Oct 30 03:24:30.935677 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.935176    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12fcc0c8-59a1-4cf7-8445-1ef34984f036-proxy-tls\" (UniqueName: \"kubernetes.io/secret/12fcc0c8-59a1-4cf7-8445-1ef34984f036-proxy-tls\") pod \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\" (UID: \"12fcc0c8-59a1-4cf7-8445-1ef34984f036\") "
Oct 30 03:24:30.935677 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.935433    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-rbac-proxy-sar-config" pod "12fcc0c8-59a1-4cf7-8445-1ef34984f036" (UID: "12fcc0c8-59a1-4cf7-8445-1ef34984f036"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:30.937945 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.937913    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-api-access-p6k7p" pod "12fcc0c8-59a1-4cf7-8445-1ef34984f036" (UID: "12fcc0c8-59a1-4cf7-8445-1ef34984f036"). InnerVolumeSpecName "kube-api-access-p6k7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:30.938481 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:30.938461    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12fcc0c8-59a1-4cf7-8445-1ef34984f036-proxy-tls" pod "12fcc0c8-59a1-4cf7-8445-1ef34984f036" (UID: "12fcc0c8-59a1-4cf7-8445-1ef34984f036"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:31.036226 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:31.036186    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:24:31.036226 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:31.036228    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6k7p\" (UniqueName: \"kubernetes.io/projected/12fcc0c8-59a1-4cf7-8445-1ef34984f036-kube-api-access-p6k7p\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:24:31.036431 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:31.036245    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12fcc0c8-59a1-4cf7-8445-1ef34984f036-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:24:31.683960 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:31.683922    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th" event={"ID":"12fcc0c8-59a1-4cf7-8445-1ef34984f036","Type":"ContainerDied","Data":"09fe8d64856503aaec84d5a426e3ee6176b62fda043f66b8876de01e42b72dbf"}
Oct 30 03:24:31.683960 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:31.683963    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"
Oct 30 03:24:31.684171 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:31.684004    4885 scope.go:122] "RemoveContainer" containerID="58dc8ba1e2abb07331a524b496335ea03fc7af5332ed7e0132f3dae16b6f207c"
Oct 30 03:24:31.692999 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:31.692982    4885 scope.go:122] "RemoveContainer" containerID="e06ff6a27d86dc7626a30dc6b5e5f2a23536dd7cd9545a151c091f5864989d1a"
Oct 30 03:24:31.729937 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:31.729909    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"]
Oct 30 03:24:31.740248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:31.740216    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/message-dumper-predictor-5d746b9b7c-752th"]
Oct 30 03:24:32.619960 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:32.619894    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:24:32.620457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:32.620261    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:24:32.625020 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:32.624996    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12fcc0c8-59a1-4cf7-8445-1ef34984f036" path="/var/lib/kubelet/pods/12fcc0c8-59a1-4cf7-8445-1ef34984f036/volumes"
Oct 30 03:24:33.197325 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:33.197278    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused"
Oct 30 03:24:35.705995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:35.705959    4885 generic.go:356] "Generic (PLEG): container finished" podID="139693d0-5632-4330-a851-dd512cc6a707" containerID="03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba" exitCode=0
Oct 30 03:24:35.706381 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:35.706038    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" event={"ID":"139693d0-5632-4330-a851-dd512cc6a707","Type":"ContainerDied","Data":"03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba"}
Oct 30 03:24:38.197167 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:38.197126    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused"
Oct 30 03:24:38.518508 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:38.518482    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:24:42.619382 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:42.619332    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:24:42.619843 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:42.619765    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:24:43.197726 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:43.197684    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused"
Oct 30 03:24:43.197958 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:43.197831    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:24:44.498342 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:44.498312    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"]
Oct 30 03:24:44.498796 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:44.498540    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="kserve-container" containerID="cri-o://b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5" gracePeriod=30
Oct 30 03:24:44.498796 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:44.498589    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="kube-rbac-proxy" containerID="cri-o://adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d" gracePeriod=30
Oct 30 03:24:44.747350 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:44.747316    4885 generic.go:356] "Generic (PLEG): container finished" podID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerID="adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d" exitCode=0
Oct 30 03:24:44.747520 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:44.747393    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" event={"ID":"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a","Type":"ContainerDied","Data":"adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d"}
Oct 30 03:24:48.197057 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:48.197011    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused"
Oct 30 03:24:48.510252 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:48.510213    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.65:8643/healthz\": dial tcp 10.132.0.65:8643: connect: connection refused"
Oct 30 03:24:48.515577 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:48.515551    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.65:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 30 03:24:51.543826 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.543805    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:24:51.605136 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.605060    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kserve-provision-location\") pod \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") "
Oct 30 03:24:51.605136 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.605118    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-proxy-tls\") pod \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") "
Oct 30 03:24:51.605331 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.605143    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-cabundle-cert\") pod \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") "
Oct 30 03:24:51.605331 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.605161    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-rbac-proxy-sar-config\") pod \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") "
Oct 30 03:24:51.605331 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.605192    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-api-access-rgvgc\" (UniqueName: \"kubernetes.io/projected/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-api-access-rgvgc\") pod \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\" (UID: \"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a\") "
Oct 30 03:24:51.605500 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.605414    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kserve-provision-location" pod "387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" (UID: "387e4eef-2f7c-463c-9ffd-4f3a09adfe7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:51.605561 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.605500    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-cabundle-cert" pod "387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" (UID: "387e4eef-2f7c-463c-9ffd-4f3a09adfe7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:51.605561 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.605512    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-rbac-proxy-sar-config" pod "387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" (UID: "387e4eef-2f7c-463c-9ffd-4f3a09adfe7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:51.607528 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.607503    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-api-access-rgvgc" pod "387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" (UID: "387e4eef-2f7c-463c-9ffd-4f3a09adfe7a"). InnerVolumeSpecName "kube-api-access-rgvgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:51.607702 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.607677    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-proxy-tls" pod "387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" (UID: "387e4eef-2f7c-463c-9ffd-4f3a09adfe7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:51.706104 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.706073    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:24:51.706104 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.706100    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:24:51.706272 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.706113    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:24:51.706272 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.706126    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:24:51.706272 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.706139    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgvgc\" (UniqueName: \"kubernetes.io/projected/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a-kube-api-access-rgvgc\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:24:51.780768 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.780736    4885 generic.go:356] "Generic (PLEG): container finished" podID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerID="b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5" exitCode=0
Oct 30 03:24:51.780943 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.780795    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"
Oct 30 03:24:51.780943 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.780798    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" event={"ID":"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a","Type":"ContainerDied","Data":"b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5"}
Oct 30 03:24:51.780943 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.780918    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr" event={"ID":"387e4eef-2f7c-463c-9ffd-4f3a09adfe7a","Type":"ContainerDied","Data":"bb83166465c8b0c01d09e9136dd5ff97ce1afd9f95b359a890757b5c6b700e73"}
Oct 30 03:24:51.780943 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.780944    4885 scope.go:122] "RemoveContainer" containerID="adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d"
Oct 30 03:24:51.789754 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.789737    4885 scope.go:122] "RemoveContainer" containerID="b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5"
Oct 30 03:24:51.797132 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.797113    4885 scope.go:122] "RemoveContainer" containerID="61b061c2b758bfc474de6ffddecb299fb4d6fc90288139f24d99dc00dc92d9b4"
Oct 30 03:24:51.804333 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.804310    4885 scope.go:122] "RemoveContainer" containerID="adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d"
Oct 30 03:24:51.804582 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:24:51.804564    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d\": container with ID starting with adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d not found: ID does not exist" containerID="adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d"
Oct 30 03:24:51.804705 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.804589    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d"} err="failed to get container status \"adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d\": rpc error: code = NotFound desc = could not find container \"adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d\": container with ID starting with adc0e3e93d34972928fab994891933442960b4b57a0079f17d31737fc88f7c5d not found: ID does not exist"
Oct 30 03:24:51.804705 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.804604    4885 scope.go:122] "RemoveContainer" containerID="b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5"
Oct 30 03:24:51.804871 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:24:51.804850    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5\": container with ID starting with b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5 not found: ID does not exist" containerID="b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5"
Oct 30 03:24:51.804935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.804882    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5"} err="failed to get container status \"b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5\": rpc error: code = NotFound desc = could not find container \"b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5\": container with ID starting with b91e39fe6eac534db961105b97ce594a95eca7cc9eeea576f64cbd8cf5d17ae5 not found: ID does not exist"
Oct 30 03:24:51.804935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.804905    4885 scope.go:122] "RemoveContainer" containerID="61b061c2b758bfc474de6ffddecb299fb4d6fc90288139f24d99dc00dc92d9b4"
Oct 30 03:24:51.805156 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:24:51.805138    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b061c2b758bfc474de6ffddecb299fb4d6fc90288139f24d99dc00dc92d9b4\": container with ID starting with 61b061c2b758bfc474de6ffddecb299fb4d6fc90288139f24d99dc00dc92d9b4 not found: ID does not exist" containerID="61b061c2b758bfc474de6ffddecb299fb4d6fc90288139f24d99dc00dc92d9b4"
Oct 30 03:24:51.805211 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.805164    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b061c2b758bfc474de6ffddecb299fb4d6fc90288139f24d99dc00dc92d9b4"} err="failed to get container status \"61b061c2b758bfc474de6ffddecb299fb4d6fc90288139f24d99dc00dc92d9b4\": rpc error: code = NotFound desc = could not find container \"61b061c2b758bfc474de6ffddecb299fb4d6fc90288139f24d99dc00dc92d9b4\": container with ID starting with 61b061c2b758bfc474de6ffddecb299fb4d6fc90288139f24d99dc00dc92d9b4 not found: ID does not exist"
Oct 30 03:24:51.826558 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.826535    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"]
Oct 30 03:24:51.830394 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:51.830376    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/sklearn-v2-mlserver-predictor-54c6b849f9-d9sjr"]
Oct 30 03:24:52.619837 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:52.619803    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 30 03:24:52.621431 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:52.620498    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:24:52.624630 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:52.624589    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" path="/var/lib/kubelet/pods/387e4eef-2f7c-463c-9ffd-4f3a09adfe7a/volumes"
Oct 30 03:24:52.625002 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:52.624990    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:24:52.625070 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:52.625062    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:24:53.197682 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:53.197640    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused"
Oct 30 03:24:54.266130 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266099    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"]
Oct 30 03:24:54.266578 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266535    4885 cpu_manager.go:414] "containerMap: removing container" podUID="12fcc0c8-59a1-4cf7-8445-1ef34984f036" containerName="kserve-container"
Oct 30 03:24:54.266578 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266553    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12fcc0c8-59a1-4cf7-8445-1ef34984f036" containerName="kserve-container"
Oct 30 03:24:54.266578 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266574    4885 cpu_manager.go:414] "containerMap: removing container" podUID="12fcc0c8-59a1-4cf7-8445-1ef34984f036" containerName="kube-rbac-proxy"
Oct 30 03:24:54.266784 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266583    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12fcc0c8-59a1-4cf7-8445-1ef34984f036" containerName="kube-rbac-proxy"
Oct 30 03:24:54.266784 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266602    4885 cpu_manager.go:414] "containerMap: removing container" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="kube-rbac-proxy"
Oct 30 03:24:54.266784 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266626    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="kube-rbac-proxy"
Oct 30 03:24:54.266784 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266647    4885 cpu_manager.go:414] "containerMap: removing container" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="kserve-container"
Oct 30 03:24:54.266784 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266654    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="kserve-container"
Oct 30 03:24:54.266784 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266663    4885 cpu_manager.go:414] "containerMap: removing container" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="storage-initializer"
Oct 30 03:24:54.266784 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266670    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="storage-initializer"
Oct 30 03:24:54.266784 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266766    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="12fcc0c8-59a1-4cf7-8445-1ef34984f036" containerName="kserve-container"
Oct 30 03:24:54.266784 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266777    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="12fcc0c8-59a1-4cf7-8445-1ef34984f036" containerName="kube-rbac-proxy"
Oct 30 03:24:54.266784 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266785    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="kube-rbac-proxy"
Oct 30 03:24:54.267158 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.266794    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="387e4eef-2f7c-463c-9ffd-4f3a09adfe7a" containerName="kserve-container"
Oct 30 03:24:54.269000 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.268982    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.282017 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.281997    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:24:54.282111 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.282021    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:24:54.282111 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.282080    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Oct 30 03:24:54.292188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.292160    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Oct 30 03:24:54.292306 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.292212    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:24:54.292306 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.292242    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:24:54.292306 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.292298    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:24:54.294480 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.294451    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"]
Oct 30 03:24:54.326248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.326227    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ba37a23-c222-4f03-8ad6-c3385a9aff67-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.326347 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.326266    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.326347 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.326298    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.326488 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.326469    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.326557 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.326517    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjr8x\" (UniqueName: \"kubernetes.io/projected/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-api-access-fjr8x\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.427935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.427908    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.428052 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.427952    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.428106 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.428073    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.428163 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.428113    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjr8x\" (UniqueName: \"kubernetes.io/projected/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-api-access-fjr8x\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.428163 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.428146    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ba37a23-c222-4f03-8ad6-c3385a9aff67-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.428325 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.428305    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.428650 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.428608    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.428741 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.428668    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.430711 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.430694    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ba37a23-c222-4f03-8ad6-c3385a9aff67-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.450081 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.450061    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjr8x\" (UniqueName: \"kubernetes.io/projected/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-api-access-fjr8x\") pod \"isvc-sklearn-runtime-predictor-5686999dc7-k654d\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.582083 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.582015    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:24:54.793462 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.793434    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:24:54.794670 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.794648    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:24:54.798710 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.798681    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:24:54.799970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.799950    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:24:54.892990 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.892850    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"]
Oct 30 03:24:54.897276 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:24:54.897253    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba37a23_c222_4f03_8ad6_c3385a9aff67.slice/crio-645585fb7d0b71b7b1b0f503da95b34c3821e3137013590d2cfc345b15374899 WatchSource:0}: Error finding container 645585fb7d0b71b7b1b0f503da95b34c3821e3137013590d2cfc345b15374899: Status 404 returned error can't find the container with id 645585fb7d0b71b7b1b0f503da95b34c3821e3137013590d2cfc345b15374899
Oct 30 03:24:54.910911 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.910890    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:24:54.955754 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.955735    4885 scope.go:122] "RemoveContainer" containerID="707de324d7f53ec62dcc5cf025560cb0471372bfa4d2368156fad87a0b046bb1"
Oct 30 03:24:54.963354 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.963333    4885 scope.go:122] "RemoveContainer" containerID="a0f082df6ed596928ad2bafef5ed012e95c67693ea848491b24d51d5872faba6"
Oct 30 03:24:54.970847 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:54.970817    4885 scope.go:122] "RemoveContainer" containerID="8d2f1540baf35f9a87d8b0afc1d9d8f0c199f2b1ee588ccb3fe2006c94bd01a4"
Oct 30 03:24:55.799944 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:55.799909    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" event={"ID":"6ba37a23-c222-4f03-8ad6-c3385a9aff67","Type":"ContainerStarted","Data":"5dfec5e48656523e16e536a4580110ae634665e3a0b4ef4deaabee377368176c"}
Oct 30 03:24:55.800327 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:55.799968    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" event={"ID":"6ba37a23-c222-4f03-8ad6-c3385a9aff67","Type":"ContainerStarted","Data":"645585fb7d0b71b7b1b0f503da95b34c3821e3137013590d2cfc345b15374899"}
Oct 30 03:24:55.921385 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:24:55.921352    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:24:58.197814 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:24:58.197768    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused"
Oct 30 03:25:00.618292 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.618265    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:25:00.680579 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.680453    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/139693d0-5632-4330-a851-dd512cc6a707-proxy-tls\" (UniqueName: \"kubernetes.io/secret/139693d0-5632-4330-a851-dd512cc6a707-proxy-tls\") pod \"139693d0-5632-4330-a851-dd512cc6a707\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") "
Oct 30 03:25:00.680579 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.680506    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/139693d0-5632-4330-a851-dd512cc6a707-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/139693d0-5632-4330-a851-dd512cc6a707-kserve-provision-location\") pod \"139693d0-5632-4330-a851-dd512cc6a707\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") "
Oct 30 03:25:00.680579 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.680538    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-kube-rbac-proxy-sar-config\") pod \"139693d0-5632-4330-a851-dd512cc6a707\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") "
Oct 30 03:25:00.680579 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.680575    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-cabundle-cert\") pod \"139693d0-5632-4330-a851-dd512cc6a707\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") "
Oct 30 03:25:00.680962 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.680640    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/139693d0-5632-4330-a851-dd512cc6a707-kube-api-access-4886z\" (UniqueName: \"kubernetes.io/projected/139693d0-5632-4330-a851-dd512cc6a707-kube-api-access-4886z\") pod \"139693d0-5632-4330-a851-dd512cc6a707\" (UID: \"139693d0-5632-4330-a851-dd512cc6a707\") "
Oct 30 03:25:00.681107 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.680967    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139693d0-5632-4330-a851-dd512cc6a707-kserve-provision-location" pod "139693d0-5632-4330-a851-dd512cc6a707" (UID: "139693d0-5632-4330-a851-dd512cc6a707"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:00.681107 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.681065    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-kube-rbac-proxy-sar-config" pod "139693d0-5632-4330-a851-dd512cc6a707" (UID: "139693d0-5632-4330-a851-dd512cc6a707"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:00.681265 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.681109    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-cabundle-cert" pod "139693d0-5632-4330-a851-dd512cc6a707" (UID: "139693d0-5632-4330-a851-dd512cc6a707"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:00.683394 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.683363    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139693d0-5632-4330-a851-dd512cc6a707-kube-api-access-4886z" pod "139693d0-5632-4330-a851-dd512cc6a707" (UID: "139693d0-5632-4330-a851-dd512cc6a707"). InnerVolumeSpecName "kube-api-access-4886z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:00.685274 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.685240    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/139693d0-5632-4330-a851-dd512cc6a707-proxy-tls" pod "139693d0-5632-4330-a851-dd512cc6a707" (UID: "139693d0-5632-4330-a851-dd512cc6a707"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:00.781798 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.781764    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/139693d0-5632-4330-a851-dd512cc6a707-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:25:00.781798 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.781797    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/139693d0-5632-4330-a851-dd512cc6a707-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:25:00.781987 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.781813    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:25:00.781987 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.781828    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/139693d0-5632-4330-a851-dd512cc6a707-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:25:00.781987 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.781842    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4886z\" (UniqueName: \"kubernetes.io/projected/139693d0-5632-4330-a851-dd512cc6a707-kube-api-access-4886z\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:25:00.822549 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.822515    4885 generic.go:356] "Generic (PLEG): container finished" podID="139693d0-5632-4330-a851-dd512cc6a707" containerID="30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef" exitCode=0
Oct 30 03:25:00.822731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.822559    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" event={"ID":"139693d0-5632-4330-a851-dd512cc6a707","Type":"ContainerDied","Data":"30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef"}
Oct 30 03:25:00.822731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.822588    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"
Oct 30 03:25:00.822731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.822649    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z" event={"ID":"139693d0-5632-4330-a851-dd512cc6a707","Type":"ContainerDied","Data":"19279a49a37a658b5a876e19c28288a31ec04ebc25cc251bd690359a6181fb19"}
Oct 30 03:25:00.822731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.822693    4885 scope.go:122] "RemoveContainer" containerID="30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef"
Oct 30 03:25:00.833107 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.833089    4885 scope.go:122] "RemoveContainer" containerID="051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4"
Oct 30 03:25:00.841283 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.841262    4885 scope.go:122] "RemoveContainer" containerID="03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba"
Oct 30 03:25:00.849074 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.849059    4885 scope.go:122] "RemoveContainer" containerID="701f4c1de9c9ff4fa409598d37aff4d08058bf6f1c0522536b1ac2ae389be362"
Oct 30 03:25:00.857012 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.856994    4885 scope.go:122] "RemoveContainer" containerID="30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef"
Oct 30 03:25:00.857328 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:25:00.857309    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef\": container with ID starting with 30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef not found: ID does not exist" containerID="30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef"
Oct 30 03:25:00.857390 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.857333    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef"} err="failed to get container status \"30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef\": rpc error: code = NotFound desc = could not find container \"30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef\": container with ID starting with 30df3ae0067ffc5561f8d3c684a6d64759af06b3cc3009e2e583c888a8f280ef not found: ID does not exist"
Oct 30 03:25:00.857390 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.857349    4885 scope.go:122] "RemoveContainer" containerID="051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4"
Oct 30 03:25:00.857570 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:25:00.857555    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4\": container with ID starting with 051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4 not found: ID does not exist" containerID="051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4"
Oct 30 03:25:00.857634 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.857574    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4"} err="failed to get container status \"051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4\": rpc error: code = NotFound desc = could not find container \"051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4\": container with ID starting with 051ee1e1e1f589583d065f4d49c01d34f4d610afbfef363aaa57bacd96b5c5d4 not found: ID does not exist"
Oct 30 03:25:00.857634 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.857585    4885 scope.go:122] "RemoveContainer" containerID="03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba"
Oct 30 03:25:00.857871 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:25:00.857855    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba\": container with ID starting with 03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba not found: ID does not exist" containerID="03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba"
Oct 30 03:25:00.857928 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.857878    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba"} err="failed to get container status \"03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba\": rpc error: code = NotFound desc = could not find container \"03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba\": container with ID starting with 03a86e5d8d48a21e241a81090bf8485f7c27c0ca783a015c1b900a8a04663fba not found: ID does not exist"
Oct 30 03:25:00.857928 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.857891    4885 scope.go:122] "RemoveContainer" containerID="701f4c1de9c9ff4fa409598d37aff4d08058bf6f1c0522536b1ac2ae389be362"
Oct 30 03:25:00.858124 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:25:00.858108    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"701f4c1de9c9ff4fa409598d37aff4d08058bf6f1c0522536b1ac2ae389be362\": container with ID starting with 701f4c1de9c9ff4fa409598d37aff4d08058bf6f1c0522536b1ac2ae389be362 not found: ID does not exist" containerID="701f4c1de9c9ff4fa409598d37aff4d08058bf6f1c0522536b1ac2ae389be362"
Oct 30 03:25:00.858174 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.858129    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701f4c1de9c9ff4fa409598d37aff4d08058bf6f1c0522536b1ac2ae389be362"} err="failed to get container status \"701f4c1de9c9ff4fa409598d37aff4d08058bf6f1c0522536b1ac2ae389be362\": rpc error: code = NotFound desc = could not find container \"701f4c1de9c9ff4fa409598d37aff4d08058bf6f1c0522536b1ac2ae389be362\": container with ID starting with 701f4c1de9c9ff4fa409598d37aff4d08058bf6f1c0522536b1ac2ae389be362 not found: ID does not exist"
Oct 30 03:25:00.874485 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.874459    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"]
Oct 30 03:25:00.877847 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:00.877826    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-logger-predictor-78555b5bb7-6j55z"]
Oct 30 03:25:01.829540 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:01.829509    4885 generic.go:356] "Generic (PLEG): container finished" podID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerID="5dfec5e48656523e16e536a4580110ae634665e3a0b4ef4deaabee377368176c" exitCode=0
Oct 30 03:25:01.829963 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:01.829584    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" event={"ID":"6ba37a23-c222-4f03-8ad6-c3385a9aff67","Type":"ContainerDied","Data":"5dfec5e48656523e16e536a4580110ae634665e3a0b4ef4deaabee377368176c"}
Oct 30 03:25:02.623758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:02.623718    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="139693d0-5632-4330-a851-dd512cc6a707" path="/var/lib/kubelet/pods/139693d0-5632-4330-a851-dd512cc6a707/volumes"
Oct 30 03:25:02.836117 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:02.836083    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" event={"ID":"6ba37a23-c222-4f03-8ad6-c3385a9aff67","Type":"ContainerStarted","Data":"8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9"}
Oct 30 03:25:02.836551 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:02.836145    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" event={"ID":"6ba37a23-c222-4f03-8ad6-c3385a9aff67","Type":"ContainerStarted","Data":"d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058"}
Oct 30 03:25:02.836551 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:02.836486    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:25:02.836551 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:02.836538    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:25:02.837895 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:02.837873    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 03:25:02.895676 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:02.894746    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" podStartSLOduration=2.890080503 podStartE2EDuration="8.894728824s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.004648321s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:02.89408802 +0000 UTC m=+1808.404257047" watchObservedRunningTime="2026-10-30 03:25:02.894728824 +0000 UTC m=+1808.404897845"
Oct 30 03:25:03.840861 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:03.840819    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 03:25:08.845858 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:08.845827    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:25:08.846324 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:08.846298    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 30 03:25:18.850822 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:25:18.850791    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:25:25.921518 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:25:25.921482    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:25:55.921577 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:25:55.921542    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:26:25.922437 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:26:25.922392    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:26:55.922407 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:26:55.922355    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:27:25.921583 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:27:25.921495    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:27:55.921901 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:27:55.921863    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:28:25.921681 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:28:25.921644    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:28:55.921838 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:28:55.921801    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:29:25.921537 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:29:25.921493    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:29:54.820846 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:29:54.820814    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:29:54.823284 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:29:54.823218    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:29:54.826268 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:29:54.826249    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:29:54.828594 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:29:54.828578    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:29:55.921789 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:29:55.921753    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:30:25.922124 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:30:25.922082    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:30:55.921679 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:30:55.921640    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:31:25.921383 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:31:25.921349    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:31:55.922602 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:31:55.922566    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:31:56.379892 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:56.379861    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"]
Oct 30 03:31:56.380385 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:56.380344    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="kserve-container" containerID="cri-o://d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058" gracePeriod=30
Oct 30 03:31:56.380728 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:56.380700    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="kube-rbac-proxy" containerID="cri-o://8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9" gracePeriod=30
Oct 30 03:31:56.551242 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:56.551208    4885 generic.go:356] "Generic (PLEG): container finished" podID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerID="8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9" exitCode=0
Oct 30 03:31:56.551408 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:56.551273    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" event={"ID":"6ba37a23-c222-4f03-8ad6-c3385a9aff67","Type":"ContainerDied","Data":"8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9"}
Oct 30 03:31:57.430189 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.430166    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:31:57.556648 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.556541    4885 generic.go:356] "Generic (PLEG): container finished" podID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerID="d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058" exitCode=0
Oct 30 03:31:57.556648 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.556607    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"
Oct 30 03:31:57.556841 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.556645    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" event={"ID":"6ba37a23-c222-4f03-8ad6-c3385a9aff67","Type":"ContainerDied","Data":"d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058"}
Oct 30 03:31:57.556841 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.556735    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d" event={"ID":"6ba37a23-c222-4f03-8ad6-c3385a9aff67","Type":"ContainerDied","Data":"645585fb7d0b71b7b1b0f503da95b34c3821e3137013590d2cfc345b15374899"}
Oct 30 03:31:57.556841 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.556763    4885 scope.go:122] "RemoveContainer" containerID="8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9"
Oct 30 03:31:57.564907 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.564888    4885 scope.go:122] "RemoveContainer" containerID="d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058"
Oct 30 03:31:57.572445 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.572420    4885 scope.go:122] "RemoveContainer" containerID="5dfec5e48656523e16e536a4580110ae634665e3a0b4ef4deaabee377368176c"
Oct 30 03:31:57.579902 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.579886    4885 scope.go:122] "RemoveContainer" containerID="8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9"
Oct 30 03:31:57.580159 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:31:57.580138    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9\": container with ID starting with 8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9 not found: ID does not exist" containerID="8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9"
Oct 30 03:31:57.580267 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.580168    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9"} err="failed to get container status \"8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9\": rpc error: code = NotFound desc = could not find container \"8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9\": container with ID starting with 8c21034184ec16e9db1b1e7d8632af47f99cf03d4fa2232199ff0b172c0ebdd9 not found: ID does not exist"
Oct 30 03:31:57.580267 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.580192    4885 scope.go:122] "RemoveContainer" containerID="d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058"
Oct 30 03:31:57.580425 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:31:57.580409    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058\": container with ID starting with d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058 not found: ID does not exist" containerID="d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058"
Oct 30 03:31:57.580492 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.580433    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058"} err="failed to get container status \"d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058\": rpc error: code = NotFound desc = could not find container \"d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058\": container with ID starting with d3765232ce91ebfddf62a0d4a948fffa476903f5b51116844d110df5ba2ae058 not found: ID does not exist"
Oct 30 03:31:57.580492 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.580455    4885 scope.go:122] "RemoveContainer" containerID="5dfec5e48656523e16e536a4580110ae634665e3a0b4ef4deaabee377368176c"
Oct 30 03:31:57.580708 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:31:57.580684    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dfec5e48656523e16e536a4580110ae634665e3a0b4ef4deaabee377368176c\": container with ID starting with 5dfec5e48656523e16e536a4580110ae634665e3a0b4ef4deaabee377368176c not found: ID does not exist" containerID="5dfec5e48656523e16e536a4580110ae634665e3a0b4ef4deaabee377368176c"
Oct 30 03:31:57.580755 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.580717    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dfec5e48656523e16e536a4580110ae634665e3a0b4ef4deaabee377368176c"} err="failed to get container status \"5dfec5e48656523e16e536a4580110ae634665e3a0b4ef4deaabee377368176c\": rpc error: code = NotFound desc = could not find container \"5dfec5e48656523e16e536a4580110ae634665e3a0b4ef4deaabee377368176c\": container with ID starting with 5dfec5e48656523e16e536a4580110ae634665e3a0b4ef4deaabee377368176c not found: ID does not exist"
Oct 30 03:31:57.597985 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.597961    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-rbac-proxy-sar-config\") pod \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") "
Oct 30 03:31:57.598077 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.598021    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ba37a23-c222-4f03-8ad6-c3385a9aff67-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ba37a23-c222-4f03-8ad6-c3385a9aff67-proxy-tls\") pod \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") "
Oct 30 03:31:57.598139 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.598077    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kserve-provision-location\") pod \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") "
Oct 30 03:31:57.598139 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.598106    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-cabundle-cert\") pod \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") "
Oct 30 03:31:57.598139 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.598123    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-api-access-fjr8x\" (UniqueName: \"kubernetes.io/projected/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-api-access-fjr8x\") pod \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\" (UID: \"6ba37a23-c222-4f03-8ad6-c3385a9aff67\") "
Oct 30 03:31:57.598418 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.598396    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-rbac-proxy-sar-config" pod "6ba37a23-c222-4f03-8ad6-c3385a9aff67" (UID: "6ba37a23-c222-4f03-8ad6-c3385a9aff67"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:57.598477 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.598439    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-cabundle-cert" pod "6ba37a23-c222-4f03-8ad6-c3385a9aff67" (UID: "6ba37a23-c222-4f03-8ad6-c3385a9aff67"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:57.600119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.600099    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba37a23-c222-4f03-8ad6-c3385a9aff67-proxy-tls" pod "6ba37a23-c222-4f03-8ad6-c3385a9aff67" (UID: "6ba37a23-c222-4f03-8ad6-c3385a9aff67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:57.600355 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.600326    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-api-access-fjr8x" pod "6ba37a23-c222-4f03-8ad6-c3385a9aff67" (UID: "6ba37a23-c222-4f03-8ad6-c3385a9aff67"). InnerVolumeSpecName "kube-api-access-fjr8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:57.625054 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.625029    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kserve-provision-location" pod "6ba37a23-c222-4f03-8ad6-c3385a9aff67" (UID: "6ba37a23-c222-4f03-8ad6-c3385a9aff67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:57.699603 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.699575    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:31:57.699603 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.699602    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:31:57.699603 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.699624    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjr8x\" (UniqueName: \"kubernetes.io/projected/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-api-access-fjr8x\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:31:57.699870 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.699634    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ba37a23-c222-4f03-8ad6-c3385a9aff67-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:31:57.699870 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.699643    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ba37a23-c222-4f03-8ad6-c3385a9aff67-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:31:57.897211 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.897180    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"]
Oct 30 03:31:57.901050 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:57.901025    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-runtime-predictor-5686999dc7-k654d"]
Oct 30 03:31:58.624018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:58.623985    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" path="/var/lib/kubelet/pods/6ba37a23-c222-4f03-8ad6-c3385a9aff67/volumes"
Oct 30 03:31:59.877083 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877052    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"]
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877349    4885 cpu_manager.go:414] "containerMap: removing container" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="storage-initializer"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877359    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="storage-initializer"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877380    4885 cpu_manager.go:414] "containerMap: removing container" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="storage-initializer"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877384    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="storage-initializer"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877392    4885 cpu_manager.go:414] "containerMap: removing container" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877397    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877405    4885 cpu_manager.go:414] "containerMap: removing container" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kube-rbac-proxy"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877410    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kube-rbac-proxy"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877415    4885 cpu_manager.go:414] "containerMap: removing container" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="kube-rbac-proxy"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877420    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="kube-rbac-proxy"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877427    4885 cpu_manager.go:414] "containerMap: removing container" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877431    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877437    4885 cpu_manager.go:414] "containerMap: removing container" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="kserve-container"
Oct 30 03:31:59.877457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877441    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="kserve-container"
Oct 30 03:31:59.878202 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877489    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kserve-container"
Oct 30 03:31:59.878202 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877496    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="kube-rbac-proxy"
Oct 30 03:31:59.878202 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877501    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="kube-rbac-proxy"
Oct 30 03:31:59.878202 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877507    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="139693d0-5632-4330-a851-dd512cc6a707" containerName="agent"
Oct 30 03:31:59.878202 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.877512    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba37a23-c222-4f03-8ad6-c3385a9aff67" containerName="kserve-container"
Oct 30 03:31:59.882282 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.882262    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:31:59.896493 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.896472    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 03:31:59.896608 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.896543    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:31:59.906552 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.906522    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:31:59.906552 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.906540    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:31:59.906751 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.906624    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:31:59.906751 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.906742    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Oct 30 03:31:59.906961 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.906945    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:31:59.908162 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:31:59.908145    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"]
Oct 30 03:32:00.017252 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.017213    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.017425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.017276    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.017425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.017375    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2c2k\" (UniqueName: \"kubernetes.io/projected/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-api-access-f2c2k\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.017511 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.017464    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.017511 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.017489    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.117846 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.117810    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.117846 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.117851    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.118103 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.117893    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.118103 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.117938    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.118103 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.117964    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2c2k\" (UniqueName: \"kubernetes.io/projected/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-api-access-f2c2k\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.118313 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.118286    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.118576 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.118549    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.118645 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.118569    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.120466 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.120447    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.139460 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.139396    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2c2k\" (UniqueName: \"kubernetes.io/projected/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-api-access-f2c2k\") pod \"isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.199331 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.199297    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:00.508410 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.508377    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"]
Oct 30 03:32:00.513803 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:32:00.513775    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813cae15_2bdc_4cc9_bc0a_953eff75a8fb.slice/crio-dc57ed1ee31c19323f7f9565a18f79a653a6c587ff44713617bd2b3057dfd78a WatchSource:0}: Error finding container dc57ed1ee31c19323f7f9565a18f79a653a6c587ff44713617bd2b3057dfd78a: Status 404 returned error can't find the container with id dc57ed1ee31c19323f7f9565a18f79a653a6c587ff44713617bd2b3057dfd78a
Oct 30 03:32:00.571369 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.571339    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" event={"ID":"813cae15-2bdc-4cc9-bc0a-953eff75a8fb","Type":"ContainerStarted","Data":"4d447a63dc580fe84712a0b19c9288d35e038649dfc0b0931b8f48575ea9c3e9"}
Oct 30 03:32:00.571467 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:00.571410    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" event={"ID":"813cae15-2bdc-4cc9-bc0a-953eff75a8fb","Type":"ContainerStarted","Data":"dc57ed1ee31c19323f7f9565a18f79a653a6c587ff44713617bd2b3057dfd78a"}
Oct 30 03:32:04.591650 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:04.591546    4885 generic.go:356] "Generic (PLEG): container finished" podID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerID="4d447a63dc580fe84712a0b19c9288d35e038649dfc0b0931b8f48575ea9c3e9" exitCode=0
Oct 30 03:32:04.591650 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:04.591634    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" event={"ID":"813cae15-2bdc-4cc9-bc0a-953eff75a8fb","Type":"ContainerDied","Data":"4d447a63dc580fe84712a0b19c9288d35e038649dfc0b0931b8f48575ea9c3e9"}
Oct 30 03:32:05.598113 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:05.598080    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" event={"ID":"813cae15-2bdc-4cc9-bc0a-953eff75a8fb","Type":"ContainerStarted","Data":"bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256"}
Oct 30 03:32:05.598521 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:05.598142    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" event={"ID":"813cae15-2bdc-4cc9-bc0a-953eff75a8fb","Type":"ContainerStarted","Data":"efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f"}
Oct 30 03:32:05.598521 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:05.598482    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:05.598639 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:05.598536    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:05.659299 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:05.659250    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" podStartSLOduration=2.995753382 podStartE2EDuration="6.659238066s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.663484684s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:32:05.648743835 +0000 UTC m=+2231.158912873" watchObservedRunningTime="2026-10-30 03:32:05.659238066 +0000 UTC m=+2231.169407121"
Oct 30 03:32:11.608300 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:11.608266    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:25.922476 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:32:25.922436    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:32:41.634453 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:41.634409    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 03:32:51.610872 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:32:51.610834    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:32:55.921965 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:32:55.921927    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:33:00.046119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:00.046053    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"]
Oct 30 03:33:00.046726 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:00.046353    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="kserve-container" containerID="cri-o://efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f" gracePeriod=30
Oct 30 03:33:00.046726 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:00.046530    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="kube-rbac-proxy" containerID="cri-o://bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256" gracePeriod=30
Oct 30 03:33:00.833361 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:00.833328    4885 generic.go:356] "Generic (PLEG): container finished" podID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerID="bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256" exitCode=0
Oct 30 03:33:00.833535 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:00.833395    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" event={"ID":"813cae15-2bdc-4cc9-bc0a-953eff75a8fb","Type":"ContainerDied","Data":"bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256"}
Oct 30 03:33:01.602850 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:01.602794    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.69:8643/healthz\": dial tcp 10.132.0.69:8643: connect: connection refused"
Oct 30 03:33:02.650810 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:02.650767    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.69:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:33:06.603004 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:06.602953    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.69:8643/healthz\": dial tcp 10.132.0.69:8643: connect: connection refused"
Oct 30 03:33:07.492109 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.492082    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:33:07.654738 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.654638    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kserve-provision-location\") pod \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") "
Oct 30 03:33:07.654738 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.654684    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-proxy-tls\" (UniqueName: \"kubernetes.io/secret/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-proxy-tls\") pod \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") "
Oct 30 03:33:07.654738 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.654733    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-cabundle-cert\") pod \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") "
Oct 30 03:33:07.655248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.654764    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-rbac-proxy-sar-config\") pod \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") "
Oct 30 03:33:07.655248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.654788    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-api-access-f2c2k\" (UniqueName: \"kubernetes.io/projected/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-api-access-f2c2k\") pod \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\" (UID: \"813cae15-2bdc-4cc9-bc0a-953eff75a8fb\") "
Oct 30 03:33:07.655248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.654954    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kserve-provision-location" pod "813cae15-2bdc-4cc9-bc0a-953eff75a8fb" (UID: "813cae15-2bdc-4cc9-bc0a-953eff75a8fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:33:07.655248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.655108    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-cabundle-cert" pod "813cae15-2bdc-4cc9-bc0a-953eff75a8fb" (UID: "813cae15-2bdc-4cc9-bc0a-953eff75a8fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:33:07.655248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.655130    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-rbac-proxy-sar-config" pod "813cae15-2bdc-4cc9-bc0a-953eff75a8fb" (UID: "813cae15-2bdc-4cc9-bc0a-953eff75a8fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:33:07.657087 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.657056    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-api-access-f2c2k" pod "813cae15-2bdc-4cc9-bc0a-953eff75a8fb" (UID: "813cae15-2bdc-4cc9-bc0a-953eff75a8fb"). InnerVolumeSpecName "kube-api-access-f2c2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:33:07.657202 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.657111    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-proxy-tls" pod "813cae15-2bdc-4cc9-bc0a-953eff75a8fb" (UID: "813cae15-2bdc-4cc9-bc0a-953eff75a8fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:33:07.755362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.755324    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:33:07.755362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.755352    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:33:07.755362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.755362    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:33:07.755362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.755370    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:33:07.755659 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.755379    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2c2k\" (UniqueName: \"kubernetes.io/projected/813cae15-2bdc-4cc9-bc0a-953eff75a8fb-kube-api-access-f2c2k\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:33:07.865959 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.865921    4885 generic.go:356] "Generic (PLEG): container finished" podID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerID="efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f" exitCode=0
Oct 30 03:33:07.866131 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.865983    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"
Oct 30 03:33:07.866131 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.865996    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" event={"ID":"813cae15-2bdc-4cc9-bc0a-953eff75a8fb","Type":"ContainerDied","Data":"efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f"}
Oct 30 03:33:07.866131 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.866069    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf" event={"ID":"813cae15-2bdc-4cc9-bc0a-953eff75a8fb","Type":"ContainerDied","Data":"dc57ed1ee31c19323f7f9565a18f79a653a6c587ff44713617bd2b3057dfd78a"}
Oct 30 03:33:07.866131 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.866107    4885 scope.go:122] "RemoveContainer" containerID="bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256"
Oct 30 03:33:07.875060 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.875043    4885 scope.go:122] "RemoveContainer" containerID="efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f"
Oct 30 03:33:07.882872 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.882855    4885 scope.go:122] "RemoveContainer" containerID="4d447a63dc580fe84712a0b19c9288d35e038649dfc0b0931b8f48575ea9c3e9"
Oct 30 03:33:07.890148 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.890133    4885 scope.go:122] "RemoveContainer" containerID="bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256"
Oct 30 03:33:07.890393 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:33:07.890376    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256\": container with ID starting with bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256 not found: ID does not exist" containerID="bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256"
Oct 30 03:33:07.890439 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.890404    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256"} err="failed to get container status \"bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256\": rpc error: code = NotFound desc = could not find container \"bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256\": container with ID starting with bef2c4fb0b048ab066de418f398337a42fc750ed1e004e48f78b0d0d45954256 not found: ID does not exist"
Oct 30 03:33:07.890439 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.890427    4885 scope.go:122] "RemoveContainer" containerID="efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f"
Oct 30 03:33:07.890665 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:33:07.890644    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f\": container with ID starting with efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f not found: ID does not exist" containerID="efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f"
Oct 30 03:33:07.890724 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.890673    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f"} err="failed to get container status \"efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f\": rpc error: code = NotFound desc = could not find container \"efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f\": container with ID starting with efebe549d6fa772285643ba7f5c3b8f54682a24fec87d401a108db682efb630f not found: ID does not exist"
Oct 30 03:33:07.890724 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.890690    4885 scope.go:122] "RemoveContainer" containerID="4d447a63dc580fe84712a0b19c9288d35e038649dfc0b0931b8f48575ea9c3e9"
Oct 30 03:33:07.890938 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:33:07.890923    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d447a63dc580fe84712a0b19c9288d35e038649dfc0b0931b8f48575ea9c3e9\": container with ID starting with 4d447a63dc580fe84712a0b19c9288d35e038649dfc0b0931b8f48575ea9c3e9 not found: ID does not exist" containerID="4d447a63dc580fe84712a0b19c9288d35e038649dfc0b0931b8f48575ea9c3e9"
Oct 30 03:33:07.890981 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.890943    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d447a63dc580fe84712a0b19c9288d35e038649dfc0b0931b8f48575ea9c3e9"} err="failed to get container status \"4d447a63dc580fe84712a0b19c9288d35e038649dfc0b0931b8f48575ea9c3e9\": rpc error: code = NotFound desc = could not find container \"4d447a63dc580fe84712a0b19c9288d35e038649dfc0b0931b8f48575ea9c3e9\": container with ID starting with 4d447a63dc580fe84712a0b19c9288d35e038649dfc0b0931b8f48575ea9c3e9 not found: ID does not exist"
Oct 30 03:33:07.911485 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.911422    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"]
Oct 30 03:33:07.921488 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:07.921458    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-runtime-predictor-ff67c5479-4jsrf"]
Oct 30 03:33:08.624193 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:08.624157    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" path="/var/lib/kubelet/pods/813cae15-2bdc-4cc9-bc0a-953eff75a8fb/volumes"
Oct 30 03:33:09.690302 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.690272    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"]
Oct 30 03:33:09.690758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.690647    4885 cpu_manager.go:414] "containerMap: removing container" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="kserve-container"
Oct 30 03:33:09.690758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.690663    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="kserve-container"
Oct 30 03:33:09.690758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.690674    4885 cpu_manager.go:414] "containerMap: removing container" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="kube-rbac-proxy"
Oct 30 03:33:09.690758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.690679    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="kube-rbac-proxy"
Oct 30 03:33:09.690758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.690693    4885 cpu_manager.go:414] "containerMap: removing container" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="storage-initializer"
Oct 30 03:33:09.690758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.690698    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="storage-initializer"
Oct 30 03:33:09.691077 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.690770    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="kserve-container"
Oct 30 03:33:09.691077 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.690778    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="813cae15-2bdc-4cc9-bc0a-953eff75a8fb" containerName="kube-rbac-proxy"
Oct 30 03:33:09.695594 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.695574    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.712638 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.708930    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:33:09.718597 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.718569    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:33:09.718597 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.718588    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Oct 30 03:33:09.718597 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.718576    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:33:09.718866 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.718603    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:33:09.718866 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.718681    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Oct 30 03:33:09.719087 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.719069    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:33:09.720213 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.720193    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"]
Oct 30 03:33:09.871539 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.871496    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.871733 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.871601    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-proxy-tls\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.871733 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.871682    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.871849 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.871730    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.871849 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.871750    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79pth\" (UniqueName: \"kubernetes.io/projected/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-api-access-79pth\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.972434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.972406    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.972658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.972439    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79pth\" (UniqueName: \"kubernetes.io/projected/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-api-access-79pth\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.972658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.972476    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.972658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.972503    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-proxy-tls\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.972658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.972633    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.973014 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.972991    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.973151 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.973132    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.973218 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.973195    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.975119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.975100    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-proxy-tls\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:09.993870 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:09.993838    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79pth\" (UniqueName: \"kubernetes.io/projected/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-api-access-79pth\") pod \"isvc-sklearn-v2-predictor-df546f4d7-x74wk\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:10.017786 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:10.017744    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:10.337543 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:10.337461    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"]
Oct 30 03:33:10.342385 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:33:10.342352    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc80e49b_fdb6_44b9_aff0_eb36cae24e59.slice/crio-0bfdee24dc0d8ef61d342dec08d7be4a531fab99d6db2eb80a642fe2418f7297 WatchSource:0}: Error finding container 0bfdee24dc0d8ef61d342dec08d7be4a531fab99d6db2eb80a642fe2418f7297: Status 404 returned error can't find the container with id 0bfdee24dc0d8ef61d342dec08d7be4a531fab99d6db2eb80a642fe2418f7297
Oct 30 03:33:10.881556 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:10.881524    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" event={"ID":"cc80e49b-fdb6-44b9-aff0-eb36cae24e59","Type":"ContainerStarted","Data":"354506426d084c596944f52a26a17fb63d52b0906c9f9cd2a61d4187ae840150"}
Oct 30 03:33:10.881974 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:10.881582    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" event={"ID":"cc80e49b-fdb6-44b9-aff0-eb36cae24e59","Type":"ContainerStarted","Data":"0bfdee24dc0d8ef61d342dec08d7be4a531fab99d6db2eb80a642fe2418f7297"}
Oct 30 03:33:14.898287 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:14.898253    4885 generic.go:356] "Generic (PLEG): container finished" podID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerID="354506426d084c596944f52a26a17fb63d52b0906c9f9cd2a61d4187ae840150" exitCode=0
Oct 30 03:33:14.898789 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:14.898327    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" event={"ID":"cc80e49b-fdb6-44b9-aff0-eb36cae24e59","Type":"ContainerDied","Data":"354506426d084c596944f52a26a17fb63d52b0906c9f9cd2a61d4187ae840150"}
Oct 30 03:33:15.904749 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:15.904715    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" event={"ID":"cc80e49b-fdb6-44b9-aff0-eb36cae24e59","Type":"ContainerStarted","Data":"62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0"}
Oct 30 03:33:15.905157 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:15.904774    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" event={"ID":"cc80e49b-fdb6-44b9-aff0-eb36cae24e59","Type":"ContainerStarted","Data":"d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c"}
Oct 30 03:33:15.905157 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:15.904963    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:15.955929 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:15.955874    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podStartSLOduration=3.178929755 podStartE2EDuration="6.955859423s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.776929668s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:33:15.955009351 +0000 UTC m=+2301.465178387" watchObservedRunningTime="2026-10-30 03:33:15.955859423 +0000 UTC m=+2301.466028449"
Oct 30 03:33:16.909584 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:16.909552    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:16.910910 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:16.910881    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 03:33:17.913490 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:17.913443    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 03:33:22.917488 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:22.917456    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:33:22.918064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:22.918039    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 03:33:25.921481 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:33:25.921446    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:33:32.918080 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:32.918044    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 03:33:42.918215 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:42.918176    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 03:33:52.918160 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:33:52.918119    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 03:33:55.921318 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:33:55.921275    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:34:02.918846 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:02.918807    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 03:34:12.918943 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:12.918900    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 03:34:22.918484 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:22.918447    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 03:34:25.921723 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:34:25.921662    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:34:26.619404 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:26.619364    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 30 03:34:36.624464 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:36.624389    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:34:40.653387 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:40.653353    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"]
Oct 30 03:34:40.653895 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:40.653655    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container" containerID="cri-o://d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c" gracePeriod=30
Oct 30 03:34:40.653895 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:40.653681    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kube-rbac-proxy" containerID="cri-o://62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0" gracePeriod=30
Oct 30 03:34:41.264951 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:41.264914    4885 generic.go:356] "Generic (PLEG): container finished" podID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerID="62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0" exitCode=0
Oct 30 03:34:41.265125 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:41.264974    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" event={"ID":"cc80e49b-fdb6-44b9-aff0-eb36cae24e59","Type":"ContainerDied","Data":"62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0"}
Oct 30 03:34:42.914246 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:42.914207    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.70:8643/healthz\": dial tcp 10.132.0.70:8643: connect: connection refused"
Oct 30 03:34:45.502798 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.502776    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:34:45.525432 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.525404    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-api-access-79pth\" (UniqueName: \"kubernetes.io/projected/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-api-access-79pth\") pod \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") "
Oct 30 03:34:45.525563 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.525442    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-cabundle-cert\") pod \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") "
Oct 30 03:34:45.525563 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.525466    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kserve-provision-location\") pod \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") "
Oct 30 03:34:45.525563 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.525538    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-rbac-proxy-sar-config\") pod \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") "
Oct 30 03:34:45.525755 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.525593    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-proxy-tls\") pod \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\" (UID: \"cc80e49b-fdb6-44b9-aff0-eb36cae24e59\") "
Oct 30 03:34:45.526058 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.526019    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kserve-provision-location" pod "cc80e49b-fdb6-44b9-aff0-eb36cae24e59" (UID: "cc80e49b-fdb6-44b9-aff0-eb36cae24e59"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:45.526234 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.526202    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-cabundle-cert" pod "cc80e49b-fdb6-44b9-aff0-eb36cae24e59" (UID: "cc80e49b-fdb6-44b9-aff0-eb36cae24e59"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:45.528715 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.526277    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-rbac-proxy-sar-config" pod "cc80e49b-fdb6-44b9-aff0-eb36cae24e59" (UID: "cc80e49b-fdb6-44b9-aff0-eb36cae24e59"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:45.532733 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.532709    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-api-access-79pth" pod "cc80e49b-fdb6-44b9-aff0-eb36cae24e59" (UID: "cc80e49b-fdb6-44b9-aff0-eb36cae24e59"). InnerVolumeSpecName "kube-api-access-79pth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:45.533375 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.533357    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-proxy-tls" pod "cc80e49b-fdb6-44b9-aff0-eb36cae24e59" (UID: "cc80e49b-fdb6-44b9-aff0-eb36cae24e59"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:34:45.627641 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.627526    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:34:45.627641 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.627567    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79pth\" (UniqueName: \"kubernetes.io/projected/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-api-access-79pth\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:34:45.627641 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.627578    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:34:45.627641 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.627586    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:34:45.627641 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:45.627596    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc80e49b-fdb6-44b9-aff0-eb36cae24e59-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:34:46.286557 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.286521    4885 generic.go:356] "Generic (PLEG): container finished" podID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerID="d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c" exitCode=0
Oct 30 03:34:46.286747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.286571    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" event={"ID":"cc80e49b-fdb6-44b9-aff0-eb36cae24e59","Type":"ContainerDied","Data":"d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c"}
Oct 30 03:34:46.286747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.286589    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"
Oct 30 03:34:46.286747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.286641    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk" event={"ID":"cc80e49b-fdb6-44b9-aff0-eb36cae24e59","Type":"ContainerDied","Data":"0bfdee24dc0d8ef61d342dec08d7be4a531fab99d6db2eb80a642fe2418f7297"}
Oct 30 03:34:46.286747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.286672    4885 scope.go:122] "RemoveContainer" containerID="62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0"
Oct 30 03:34:46.295837 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.295818    4885 scope.go:122] "RemoveContainer" containerID="d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c"
Oct 30 03:34:46.305891 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.305873    4885 scope.go:122] "RemoveContainer" containerID="354506426d084c596944f52a26a17fb63d52b0906c9f9cd2a61d4187ae840150"
Oct 30 03:34:46.313436 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.313418    4885 scope.go:122] "RemoveContainer" containerID="62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0"
Oct 30 03:34:46.313707 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:34:46.313685    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0\": container with ID starting with 62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0 not found: ID does not exist" containerID="62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0"
Oct 30 03:34:46.313795 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.313713    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0"} err="failed to get container status \"62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0\": rpc error: code = NotFound desc = could not find container \"62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0\": container with ID starting with 62831b5349b86a8765c076a38f4a30a84885dff989040a906fe246fb9dc4dce0 not found: ID does not exist"
Oct 30 03:34:46.313795 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.313729    4885 scope.go:122] "RemoveContainer" containerID="d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c"
Oct 30 03:34:46.313995 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:34:46.313974    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c\": container with ID starting with d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c not found: ID does not exist" containerID="d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c"
Oct 30 03:34:46.314063 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.314004    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c"} err="failed to get container status \"d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c\": rpc error: code = NotFound desc = could not find container \"d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c\": container with ID starting with d52170a48a5e7191f28ae219544fe40ccb675af2fc07e644063e919efa0a8a1c not found: ID does not exist"
Oct 30 03:34:46.314063 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.314026    4885 scope.go:122] "RemoveContainer" containerID="354506426d084c596944f52a26a17fb63d52b0906c9f9cd2a61d4187ae840150"
Oct 30 03:34:46.314282 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:34:46.314264    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354506426d084c596944f52a26a17fb63d52b0906c9f9cd2a61d4187ae840150\": container with ID starting with 354506426d084c596944f52a26a17fb63d52b0906c9f9cd2a61d4187ae840150 not found: ID does not exist" containerID="354506426d084c596944f52a26a17fb63d52b0906c9f9cd2a61d4187ae840150"
Oct 30 03:34:46.314337 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.314289    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354506426d084c596944f52a26a17fb63d52b0906c9f9cd2a61d4187ae840150"} err="failed to get container status \"354506426d084c596944f52a26a17fb63d52b0906c9f9cd2a61d4187ae840150\": rpc error: code = NotFound desc = could not find container \"354506426d084c596944f52a26a17fb63d52b0906c9f9cd2a61d4187ae840150\": container with ID starting with 354506426d084c596944f52a26a17fb63d52b0906c9f9cd2a61d4187ae840150 not found: ID does not exist"
Oct 30 03:34:46.342470 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.342447    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"]
Oct 30 03:34:46.346154 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.346136    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-predictor-df546f4d7-x74wk"]
Oct 30 03:34:46.624427 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:46.624354    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" path="/var/lib/kubelet/pods/cc80e49b-fdb6-44b9-aff0-eb36cae24e59/volumes"
Oct 30 03:34:48.240230 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.240192    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"]
Oct 30 03:34:48.240756 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.240653    4885 cpu_manager.go:414] "containerMap: removing container" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kube-rbac-proxy"
Oct 30 03:34:48.240756 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.240673    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kube-rbac-proxy"
Oct 30 03:34:48.240756 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.240688    4885 cpu_manager.go:414] "containerMap: removing container" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container"
Oct 30 03:34:48.240756 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.240696    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container"
Oct 30 03:34:48.240756 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.240717    4885 cpu_manager.go:414] "containerMap: removing container" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="storage-initializer"
Oct 30 03:34:48.240756 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.240725    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="storage-initializer"
Oct 30 03:34:48.241152 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.240817    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kube-rbac-proxy"
Oct 30 03:34:48.241152 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.240829    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc80e49b-fdb6-44b9-aff0-eb36cae24e59" containerName="kserve-container"
Oct 30 03:34:48.245758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.245737    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.258452 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.258435    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:34:48.258594 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.258574    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:34:48.258691 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.258586    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Oct 30 03:34:48.268645 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.268592    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:34:48.268794 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.268774    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Oct 30 03:34:48.268914 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.268841    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:34:48.268914 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.268854    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:34:48.270665 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.270646    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"]
Oct 30 03:34:48.349268 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.349238    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.349434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.349277    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a4400ff-ef58-4740-b8fe-bcce0158c466-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.349434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.349297    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7492\" (UniqueName: \"kubernetes.io/projected/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-api-access-s7492\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.349434 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.349417    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a4400ff-ef58-4740-b8fe-bcce0158c466-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.349570 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.349461    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.450754 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.450720    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.450949 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.450771    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a4400ff-ef58-4740-b8fe-bcce0158c466-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.450949 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.450803    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7492\" (UniqueName: \"kubernetes.io/projected/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-api-access-s7492\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.450949 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.450841    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a4400ff-ef58-4740-b8fe-bcce0158c466-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.450949 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.450888    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.451225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.451202    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a4400ff-ef58-4740-b8fe-bcce0158c466-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.451473 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.451449    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.451563 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.451546    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.453520 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.453505    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a4400ff-ef58-4740-b8fe-bcce0158c466-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.472302 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.472274    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7492\" (UniqueName: \"kubernetes.io/projected/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-api-access-s7492\") pod \"isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.562105 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.562038    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:48.868497 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:48.868426    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"]
Oct 30 03:34:48.873482 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:34:48.873452    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4400ff_ef58_4740_b8fe_bcce0158c466.slice/crio-4889d22c626dc1fd99bc0efa510765e6312e14e8f966fbb0f34cc8a65cc4f7f4 WatchSource:0}: Error finding container 4889d22c626dc1fd99bc0efa510765e6312e14e8f966fbb0f34cc8a65cc4f7f4: Status 404 returned error can't find the container with id 4889d22c626dc1fd99bc0efa510765e6312e14e8f966fbb0f34cc8a65cc4f7f4
Oct 30 03:34:49.304680 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:49.304637    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" event={"ID":"0a4400ff-ef58-4740-b8fe-bcce0158c466","Type":"ContainerStarted","Data":"2be0b89f4d6eb3a438e3ca6a36d5c82d93ebf075dc5f133227794235077d9681"}
Oct 30 03:34:49.305114 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:49.304719    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" event={"ID":"0a4400ff-ef58-4740-b8fe-bcce0158c466","Type":"ContainerStarted","Data":"4889d22c626dc1fd99bc0efa510765e6312e14e8f966fbb0f34cc8a65cc4f7f4"}
Oct 30 03:34:53.325030 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:53.324994    4885 generic.go:356] "Generic (PLEG): container finished" podID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerID="2be0b89f4d6eb3a438e3ca6a36d5c82d93ebf075dc5f133227794235077d9681" exitCode=0
Oct 30 03:34:53.325030 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:53.325034    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" event={"ID":"0a4400ff-ef58-4740-b8fe-bcce0158c466","Type":"ContainerDied","Data":"2be0b89f4d6eb3a438e3ca6a36d5c82d93ebf075dc5f133227794235077d9681"}
Oct 30 03:34:54.331889 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:54.331846    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" event={"ID":"0a4400ff-ef58-4740-b8fe-bcce0158c466","Type":"ContainerStarted","Data":"baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d"}
Oct 30 03:34:54.332345 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:54.331924    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" event={"ID":"0a4400ff-ef58-4740-b8fe-bcce0158c466","Type":"ContainerStarted","Data":"0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e"}
Oct 30 03:34:54.332345 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:54.332255    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:54.332439 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:54.332391    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:34:54.333888 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:54.333864    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 03:34:54.383409 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:54.383359    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podStartSLOduration=2.634042009 podStartE2EDuration="6.383346765s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.749304756s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:54.381948229 +0000 UTC m=+2399.892117266" watchObservedRunningTime="2026-10-30 03:34:54.383346765 +0000 UTC m=+2399.893515792"
Oct 30 03:34:54.848546 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:54.848432    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:34:54.867503 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:54.851760    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:34:54.867503 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:54.854160    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:34:54.867503 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:54.863731    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:34:55.337257 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:34:55.337216    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 03:34:55.920944 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:34:55.920910    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:35:00.342659 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:35:00.342606    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:35:00.343258 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:35:00.343186    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 03:35:10.343742 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:35:10.343699    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 03:35:20.343803 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:35:20.343758    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 03:35:25.922256 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:35:25.922220    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:35:30.343127 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:35:30.343086    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 03:35:40.344155 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:35:40.344114    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 03:35:50.343697 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:35:50.343656    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 03:35:55.921688 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:35:55.921652    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:36:00.343449 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:00.343412    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 03:36:07.619855 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:07.619806    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 30 03:36:17.620790 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:17.620757    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:36:25.921353 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:36:25.921313    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:36:29.181139 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:29.181094    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"]
Oct 30 03:36:29.181663 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:29.181393    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container" containerID="cri-o://0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e" gracePeriod=30
Oct 30 03:36:29.181663 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:29.181427    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kube-rbac-proxy" containerID="cri-o://baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d" gracePeriod=30
Oct 30 03:36:29.729054 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:29.729020    4885 generic.go:356] "Generic (PLEG): container finished" podID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerID="baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d" exitCode=0
Oct 30 03:36:29.729234 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:29.729062    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" event={"ID":"0a4400ff-ef58-4740-b8fe-bcce0158c466","Type":"ContainerDied","Data":"baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d"}
Oct 30 03:36:30.338181 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:30.338134    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.71:8643/healthz\": dial tcp 10.132.0.71:8643: connect: connection refused"
Oct 30 03:36:34.526989 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.526959    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:36:34.606871 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.606784    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4400ff-ef58-4740-b8fe-bcce0158c466-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a4400ff-ef58-4740-b8fe-bcce0158c466-kserve-provision-location\") pod \"0a4400ff-ef58-4740-b8fe-bcce0158c466\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") "
Oct 30 03:36:34.606871 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.606848    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-api-access-s7492\" (UniqueName: \"kubernetes.io/projected/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-api-access-s7492\") pod \"0a4400ff-ef58-4740-b8fe-bcce0158c466\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") "
Oct 30 03:36:34.606871 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.606866    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-rbac-proxy-sar-config\") pod \"0a4400ff-ef58-4740-b8fe-bcce0158c466\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") "
Oct 30 03:36:34.607321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.606889    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a4400ff-ef58-4740-b8fe-bcce0158c466-proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a4400ff-ef58-4740-b8fe-bcce0158c466-proxy-tls\") pod \"0a4400ff-ef58-4740-b8fe-bcce0158c466\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") "
Oct 30 03:36:34.607321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.606916    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-cabundle-cert\") pod \"0a4400ff-ef58-4740-b8fe-bcce0158c466\" (UID: \"0a4400ff-ef58-4740-b8fe-bcce0158c466\") "
Oct 30 03:36:34.607321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.607150    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4400ff-ef58-4740-b8fe-bcce0158c466-kserve-provision-location" pod "0a4400ff-ef58-4740-b8fe-bcce0158c466" (UID: "0a4400ff-ef58-4740-b8fe-bcce0158c466"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:36:34.607321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.607283    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-rbac-proxy-sar-config" pod "0a4400ff-ef58-4740-b8fe-bcce0158c466" (UID: "0a4400ff-ef58-4740-b8fe-bcce0158c466"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:36:34.607497 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.607329    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-cabundle-cert" pod "0a4400ff-ef58-4740-b8fe-bcce0158c466" (UID: "0a4400ff-ef58-4740-b8fe-bcce0158c466"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:36:34.609119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.609093    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4400ff-ef58-4740-b8fe-bcce0158c466-proxy-tls" pod "0a4400ff-ef58-4740-b8fe-bcce0158c466" (UID: "0a4400ff-ef58-4740-b8fe-bcce0158c466"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:36:34.609238 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.609211    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-api-access-s7492" pod "0a4400ff-ef58-4740-b8fe-bcce0158c466" (UID: "0a4400ff-ef58-4740-b8fe-bcce0158c466"). InnerVolumeSpecName "kube-api-access-s7492". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:36:34.708056 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.708019    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7492\" (UniqueName: \"kubernetes.io/projected/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-api-access-s7492\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:36:34.708056 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.708051    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:36:34.708056 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.708061    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a4400ff-ef58-4740-b8fe-bcce0158c466-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:36:34.708056 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.708069    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a4400ff-ef58-4740-b8fe-bcce0158c466-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:36:34.708426 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.708077    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a4400ff-ef58-4740-b8fe-bcce0158c466-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:36:34.751596 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.751561    4885 generic.go:356] "Generic (PLEG): container finished" podID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerID="0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e" exitCode=0
Oct 30 03:36:34.751792 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.751648    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"
Oct 30 03:36:34.751792 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.751644    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" event={"ID":"0a4400ff-ef58-4740-b8fe-bcce0158c466","Type":"ContainerDied","Data":"0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e"}
Oct 30 03:36:34.751792 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.751769    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5" event={"ID":"0a4400ff-ef58-4740-b8fe-bcce0158c466","Type":"ContainerDied","Data":"4889d22c626dc1fd99bc0efa510765e6312e14e8f966fbb0f34cc8a65cc4f7f4"}
Oct 30 03:36:34.751923 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.751794    4885 scope.go:122] "RemoveContainer" containerID="baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d"
Oct 30 03:36:34.762523 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.762499    4885 scope.go:122] "RemoveContainer" containerID="0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e"
Oct 30 03:36:34.770825 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.770805    4885 scope.go:122] "RemoveContainer" containerID="2be0b89f4d6eb3a438e3ca6a36d5c82d93ebf075dc5f133227794235077d9681"
Oct 30 03:36:34.778783 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.778760    4885 scope.go:122] "RemoveContainer" containerID="baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d"
Oct 30 03:36:34.779116 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:36:34.779096    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d\": container with ID starting with baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d not found: ID does not exist" containerID="baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d"
Oct 30 03:36:34.779193 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.779131    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d"} err="failed to get container status \"baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d\": rpc error: code = NotFound desc = could not find container \"baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d\": container with ID starting with baf8ff91b97a3cac1b193e214013ebbc82c1231886dba283aeb67674eb44209d not found: ID does not exist"
Oct 30 03:36:34.779193 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.779156    4885 scope.go:122] "RemoveContainer" containerID="0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e"
Oct 30 03:36:34.779440 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:36:34.779421    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e\": container with ID starting with 0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e not found: ID does not exist" containerID="0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e"
Oct 30 03:36:34.779491 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.779449    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e"} err="failed to get container status \"0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e\": rpc error: code = NotFound desc = could not find container \"0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e\": container with ID starting with 0f4fabff5a1b39679533b51781229a278742e3870473d3016a7668a48318872e not found: ID does not exist"
Oct 30 03:36:34.779491 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.779469    4885 scope.go:122] "RemoveContainer" containerID="2be0b89f4d6eb3a438e3ca6a36d5c82d93ebf075dc5f133227794235077d9681"
Oct 30 03:36:34.779761 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:36:34.779744    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be0b89f4d6eb3a438e3ca6a36d5c82d93ebf075dc5f133227794235077d9681\": container with ID starting with 2be0b89f4d6eb3a438e3ca6a36d5c82d93ebf075dc5f133227794235077d9681 not found: ID does not exist" containerID="2be0b89f4d6eb3a438e3ca6a36d5c82d93ebf075dc5f133227794235077d9681"
Oct 30 03:36:34.779819 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.779767    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be0b89f4d6eb3a438e3ca6a36d5c82d93ebf075dc5f133227794235077d9681"} err="failed to get container status \"2be0b89f4d6eb3a438e3ca6a36d5c82d93ebf075dc5f133227794235077d9681\": rpc error: code = NotFound desc = could not find container \"2be0b89f4d6eb3a438e3ca6a36d5c82d93ebf075dc5f133227794235077d9681\": container with ID starting with 2be0b89f4d6eb3a438e3ca6a36d5c82d93ebf075dc5f133227794235077d9681 not found: ID does not exist"
Oct 30 03:36:34.799407 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.799327    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"]
Oct 30 03:36:34.804100 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:34.804074    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-v2-mixed-predictor-64f8bbd4fd-ft5r5"]
Oct 30 03:36:36.624041 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:36:36.624012    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" path="/var/lib/kubelet/pods/0a4400ff-ef58-4740-b8fe-bcce0158c466/volumes"
Oct 30 03:36:55.921823 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:36:55.921787    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:37:25.921168 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:37:25.921130    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:37:55.921119 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:37:55.921079    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:38:08.140688 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.140647    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"]
Oct 30 03:38:08.141134 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.140955    4885 cpu_manager.go:414] "containerMap: removing container" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container"
Oct 30 03:38:08.141134 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.141006    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container"
Oct 30 03:38:08.141134 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.141033    4885 cpu_manager.go:414] "containerMap: removing container" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kube-rbac-proxy"
Oct 30 03:38:08.141134 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.141039    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kube-rbac-proxy"
Oct 30 03:38:08.141134 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.141054    4885 cpu_manager.go:414] "containerMap: removing container" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="storage-initializer"
Oct 30 03:38:08.141134 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.141061    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="storage-initializer"
Oct 30 03:38:08.141134 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.141121    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kube-rbac-proxy"
Oct 30 03:38:08.141134 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.141133    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4400ff-ef58-4740-b8fe-bcce0158c466" containerName="kserve-container"
Oct 30 03:38:08.144228 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.144211    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.153206 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.153185    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"]
Oct 30 03:38:08.156444 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.156428    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.161998 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.161978    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:38:08.161998 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.161977    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:38:08.162190 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.162159    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:38:08.162260 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.162242    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:38:08.162474 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.162281    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:38:08.162474 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.162332    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Oct 30 03:38:08.169050 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.169033    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Oct 30 03:38:08.184644 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.184580    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"]
Oct 30 03:38:08.185828 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.185806    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"]
Oct 30 03:38:08.227798 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.227769    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-cabundle-cert\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.227943 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.227804    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.227943 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.227824    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd750cb4-4954-4cc5-929a-d40be50a33ac-kserve-provision-location\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.227943 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.227887    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.228044 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.227964    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c7b6b6e-f217-4f26-a720-3d905d6bb261-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.228044 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.227993    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.228044 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.228019    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd750cb4-4954-4cc5-929a-d40be50a33ac-proxy-tls\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.228139 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.228119    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xp5f\" (UniqueName: \"kubernetes.io/projected/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-api-access-5xp5f\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.228173 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.228155    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brtmz\" (UniqueName: \"kubernetes.io/projected/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-api-access-brtmz\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.228225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.228212    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.260451 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.260424    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-create-predictor-serving-cert\""
Oct 30 03:38:08.328521 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.328493    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.328698 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.328531    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd750cb4-4954-4cc5-929a-d40be50a33ac-proxy-tls\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.328698 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.328657    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xp5f\" (UniqueName: \"kubernetes.io/projected/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-api-access-5xp5f\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.328830 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.328706    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brtmz\" (UniqueName: \"kubernetes.io/projected/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-api-access-brtmz\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.328830 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.328751    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.328830 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.328786    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-cabundle-cert\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.328830 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.328819    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.329036 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.328848    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd750cb4-4954-4cc5-929a-d40be50a33ac-kserve-provision-location\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.329036 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.328882    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.329036 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.328918    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c7b6b6e-f217-4f26-a720-3d905d6bb261-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.329254 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.329229    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.329323 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.329294    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd750cb4-4954-4cc5-929a-d40be50a33ac-kserve-provision-location\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.329382 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.329298    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.329488 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.329471    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.329637 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.329594    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-cabundle-cert\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.329886 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.329834    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.331400 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.331370    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd750cb4-4954-4cc5-929a-d40be50a33ac-proxy-tls\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.331489 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.331476    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c7b6b6e-f217-4f26-a720-3d905d6bb261-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.350260 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.350237    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xp5f\" (UniqueName: \"kubernetes.io/projected/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-api-access-5xp5f\") pod \"isvc-canary-create-v2-predictor-7c47557b88-fvqbg\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.350596 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.350575    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brtmz\" (UniqueName: \"kubernetes.io/projected/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-api-access-brtmz\") pod \"isvc-canary-create-predictor-6764465dcc-sq57k\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.462526 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.462498    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:08.473250 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.473216    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:08.771867 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.771768    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"]
Oct 30 03:38:08.774108 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:08.774082    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"]
Oct 30 03:38:08.775635 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:38:08.775592    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd750cb4_4954_4cc5_929a_d40be50a33ac.slice/crio-e54d07eb1063a287c5f3252d44412b9804274255e4d8cf0ec611abd3b0995bdd WatchSource:0}: Error finding container e54d07eb1063a287c5f3252d44412b9804274255e4d8cf0ec611abd3b0995bdd: Status 404 returned error can't find the container with id e54d07eb1063a287c5f3252d44412b9804274255e4d8cf0ec611abd3b0995bdd
Oct 30 03:38:08.785791 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:38:08.785763    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7b6b6e_f217_4f26_a720_3d905d6bb261.slice/crio-348d05279cc0715420bea665a6392e819b658d2dba6fcad2db4e93ca0003ee15 WatchSource:0}: Error finding container 348d05279cc0715420bea665a6392e819b658d2dba6fcad2db4e93ca0003ee15: Status 404 returned error can't find the container with id 348d05279cc0715420bea665a6392e819b658d2dba6fcad2db4e93ca0003ee15
Oct 30 03:38:09.147818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:09.147725    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" event={"ID":"1c7b6b6e-f217-4f26-a720-3d905d6bb261","Type":"ContainerStarted","Data":"94fdb1a8aad60fa80753af160a0c9871c81af2f99432eb65f3a5983b409424cb"}
Oct 30 03:38:09.147818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:09.147795    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" event={"ID":"1c7b6b6e-f217-4f26-a720-3d905d6bb261","Type":"ContainerStarted","Data":"348d05279cc0715420bea665a6392e819b658d2dba6fcad2db4e93ca0003ee15"}
Oct 30 03:38:09.149560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:09.149523    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" event={"ID":"fd750cb4-4954-4cc5-929a-d40be50a33ac","Type":"ContainerStarted","Data":"c72bffef4de53d4393a4f3e341eb8a346846a6acfef0e72c0a1c80a597f22e03"}
Oct 30 03:38:09.149704 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:09.149574    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" event={"ID":"fd750cb4-4954-4cc5-929a-d40be50a33ac","Type":"ContainerStarted","Data":"e54d07eb1063a287c5f3252d44412b9804274255e4d8cf0ec611abd3b0995bdd"}
Oct 30 03:38:13.169936 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:13.169894    4885 generic.go:356] "Generic (PLEG): container finished" podID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerID="c72bffef4de53d4393a4f3e341eb8a346846a6acfef0e72c0a1c80a597f22e03" exitCode=0
Oct 30 03:38:13.170383 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:13.169971    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" event={"ID":"fd750cb4-4954-4cc5-929a-d40be50a33ac","Type":"ContainerDied","Data":"c72bffef4de53d4393a4f3e341eb8a346846a6acfef0e72c0a1c80a597f22e03"}
Oct 30 03:38:13.171630 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:13.171591    4885 generic.go:356] "Generic (PLEG): container finished" podID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerID="94fdb1a8aad60fa80753af160a0c9871c81af2f99432eb65f3a5983b409424cb" exitCode=0
Oct 30 03:38:13.171737 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:13.171648    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" event={"ID":"1c7b6b6e-f217-4f26-a720-3d905d6bb261","Type":"ContainerDied","Data":"94fdb1a8aad60fa80753af160a0c9871c81af2f99432eb65f3a5983b409424cb"}
Oct 30 03:38:14.178209 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:14.178170    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" event={"ID":"1c7b6b6e-f217-4f26-a720-3d905d6bb261","Type":"ContainerStarted","Data":"60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b"}
Oct 30 03:38:14.178636 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:14.178239    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" event={"ID":"1c7b6b6e-f217-4f26-a720-3d905d6bb261","Type":"ContainerStarted","Data":"411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd"}
Oct 30 03:38:14.178636 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:14.178534    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:14.178778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:14.178646    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:14.179913 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:14.179887    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 03:38:14.180418 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:14.180399    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" event={"ID":"fd750cb4-4954-4cc5-929a-d40be50a33ac","Type":"ContainerStarted","Data":"7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44"}
Oct 30 03:38:14.180503 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:14.180442    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" event={"ID":"fd750cb4-4954-4cc5-929a-d40be50a33ac","Type":"ContainerStarted","Data":"a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f"}
Oct 30 03:38:14.180640 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:14.180609    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:14.180720 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:14.180673    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:14.181743 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:14.181719    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 03:38:14.219639 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:14.219583    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podStartSLOduration=2.319511001 podStartE2EDuration="6.219571317s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.900060316s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:38:14.217937332 +0000 UTC m=+2599.728106358" watchObservedRunningTime="2026-10-30 03:38:14.219571317 +0000 UTC m=+2599.729740342"
Oct 30 03:38:14.257949 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:14.257902    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podStartSLOduration=2.4288827619999998 podStartE2EDuration="6.25788937s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.829006608s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:38:14.256865689 +0000 UTC m=+2599.767034715" watchObservedRunningTime="2026-10-30 03:38:14.25788937 +0000 UTC m=+2599.768058393"
Oct 30 03:38:15.184597 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:15.184553    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 03:38:15.185016 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:15.184690    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 03:38:20.189346 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:20.189314    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:38:20.189812 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:20.189738    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:38:20.189812 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:20.189770    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 03:38:20.190130 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:20.190102    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 03:38:25.922378 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:38:25.922340    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:38:30.190665 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:30.190598    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 03:38:30.191178 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:30.190598    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 03:38:40.190693 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:40.190649    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 03:38:40.191128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:40.190661    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 03:38:48.815698 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.815661    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"]
Oct 30 03:38:48.820555 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.820534    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:48.843433 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.843409    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:38:48.843609 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.843482    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:38:48.843609 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.843522    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:38:48.843796 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.843565    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Oct 30 03:38:48.843796 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.843635    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-tensorflow-predictor-serving-cert\""
Oct 30 03:38:48.843796 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.843670    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:38:48.843796 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.843727    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:38:48.845245 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.845226    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"]
Oct 30 03:38:48.968322 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.968283    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa95de7c-cf78-4c69-b170-65b79b226eeb-kserve-provision-location\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:48.968322 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.968327    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa95de7c-cf78-4c69-b170-65b79b226eeb-proxy-tls\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:48.968584 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.968358    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:48.968584 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.968488    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbsq4\" (UniqueName: \"kubernetes.io/projected/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-api-access-xbsq4\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:48.968584 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:48.968530    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-cabundle-cert\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:49.070056 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:49.069975    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa95de7c-cf78-4c69-b170-65b79b226eeb-kserve-provision-location\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:49.070056 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:49.070021    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa95de7c-cf78-4c69-b170-65b79b226eeb-proxy-tls\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:49.070237 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:49.070141    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:49.070237 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:49.070192    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbsq4\" (UniqueName: \"kubernetes.io/projected/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-api-access-xbsq4\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:49.070309 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:49.070241    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-cabundle-cert\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:49.070398 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:49.070374    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa95de7c-cf78-4c69-b170-65b79b226eeb-kserve-provision-location\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:49.070954 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:49.070931    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:49.071070 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:49.070931    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-cabundle-cert\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:49.072807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:49.072783    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa95de7c-cf78-4c69-b170-65b79b226eeb-proxy-tls\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:49.091785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:49.091757    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbsq4\" (UniqueName: \"kubernetes.io/projected/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-api-access-xbsq4\") pod \"isvc-tensorflow-predictor-649b8d645-2swdf\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:49.138770 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:49.138738    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:49.444170 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:49.444016    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"]
Oct 30 03:38:49.447114 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:38:49.447080    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa95de7c_cf78_4c69_b170_65b79b226eeb.slice/crio-778aed06f4cebab2ac709f71ecb9efa0887819dc3defd38472bc30230b9655d9 WatchSource:0}: Error finding container 778aed06f4cebab2ac709f71ecb9efa0887819dc3defd38472bc30230b9655d9: Status 404 returned error can't find the container with id 778aed06f4cebab2ac709f71ecb9efa0887819dc3defd38472bc30230b9655d9
Oct 30 03:38:50.190274 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:50.190232    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 03:38:50.190702 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:50.190233    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 03:38:50.337702 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:50.337662    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" event={"ID":"fa95de7c-cf78-4c69-b170-65b79b226eeb","Type":"ContainerStarted","Data":"a643f9e73530a9e7928a587d8b6a50cbe3510d0411797cb724bbdde9a2e3dc3e"}
Oct 30 03:38:50.337879 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:50.337744    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" event={"ID":"fa95de7c-cf78-4c69-b170-65b79b226eeb","Type":"ContainerStarted","Data":"778aed06f4cebab2ac709f71ecb9efa0887819dc3defd38472bc30230b9655d9"}
Oct 30 03:38:55.363101 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:55.363022    4885 generic.go:356] "Generic (PLEG): container finished" podID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerID="a643f9e73530a9e7928a587d8b6a50cbe3510d0411797cb724bbdde9a2e3dc3e" exitCode=0
Oct 30 03:38:55.363497 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:55.363094    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" event={"ID":"fa95de7c-cf78-4c69-b170-65b79b226eeb","Type":"ContainerDied","Data":"a643f9e73530a9e7928a587d8b6a50cbe3510d0411797cb724bbdde9a2e3dc3e"}
Oct 30 03:38:55.364373 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:55.364358    4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:38:55.932723 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:38:55.932678    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:38:59.386391 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:59.386313    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" event={"ID":"fa95de7c-cf78-4c69-b170-65b79b226eeb","Type":"ContainerStarted","Data":"12dfae05266722f7b3b956fda611dcc22ee34e0850199e94e2733bc70f321f4a"}
Oct 30 03:38:59.386391 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:59.386379    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" event={"ID":"fa95de7c-cf78-4c69-b170-65b79b226eeb","Type":"ContainerStarted","Data":"2dbcbbc630ba60ac9aec1565ef50edc3b9065041a0a64dfba978e2568bf60993"}
Oct 30 03:38:59.386973 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:59.386558    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:38:59.438685 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:38:59.438593    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" podStartSLOduration=2.166158768 podStartE2EDuration="11.438572298s" totalImagesPullingTime="3.736587061s" totalInitContainerRuntime="5.535826469s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:38:59.438017821 +0000 UTC m=+2644.948186887" watchObservedRunningTime="2026-10-30 03:38:59.438572298 +0000 UTC m=+2644.948741325"
Oct 30 03:39:00.190170 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:00.190128    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 03:39:00.190352 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:00.190129    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 03:39:00.390759 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:00.390728    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:39:00.391912 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:00.391885    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 03:39:01.394951 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:01.394909    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 03:39:06.400096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:06.400020    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:39:06.400585 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:06.400557    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 30 03:39:10.190474 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:10.190430    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 03:39:10.190895 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:10.190430    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 03:39:16.401297 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:16.401266    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:39:20.190276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:20.190236    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 03:39:20.190675 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:20.190240    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 03:39:25.922976 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:39:25.922940    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:39:30.189889 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:30.189845    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 03:39:30.190432 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:30.190267    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 03:39:30.220833 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:30.220802    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"]
Oct 30 03:39:30.221072 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:30.221047    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kserve-container" containerID="cri-o://2dbcbbc630ba60ac9aec1565ef50edc3b9065041a0a64dfba978e2568bf60993" gracePeriod=30
Oct 30 03:39:30.221149 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:30.221082    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kube-rbac-proxy" containerID="cri-o://12dfae05266722f7b3b956fda611dcc22ee34e0850199e94e2733bc70f321f4a" gracePeriod=30
Oct 30 03:39:30.525868 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:30.525835    4885 generic.go:356] "Generic (PLEG): container finished" podID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerID="12dfae05266722f7b3b956fda611dcc22ee34e0850199e94e2733bc70f321f4a" exitCode=0
Oct 30 03:39:30.526060 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:30.525912    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" event={"ID":"fa95de7c-cf78-4c69-b170-65b79b226eeb","Type":"ContainerDied","Data":"12dfae05266722f7b3b956fda611dcc22ee34e0850199e94e2733bc70f321f4a"}
Oct 30 03:39:30.619146 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:30.619104    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 30 03:39:31.395965 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:31.395917    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.74:8643/healthz\": dial tcp 10.132.0.74:8643: connect: connection refused"
Oct 30 03:39:36.395196 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:36.395155    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.74:8643/healthz\": dial tcp 10.132.0.74:8643: connect: connection refused"
Oct 30 03:39:40.191746 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:40.191716    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:39:40.624092 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:40.624062    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:39:40.633276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:40.633252    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"]
Oct 30 03:39:41.395382 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:41.395341    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.74:8643/healthz\": dial tcp 10.132.0.74:8643: connect: connection refused"
Oct 30 03:39:41.395792 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:41.395479    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:39:41.575929 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:41.575896    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container" containerID="cri-o://a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f" gracePeriod=30
Oct 30 03:39:41.576107 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:41.575937    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kube-rbac-proxy" containerID="cri-o://7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44" gracePeriod=30
Oct 30 03:39:42.582581 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:42.582549    4885 generic.go:356] "Generic (PLEG): container finished" podID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerID="7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44" exitCode=0
Oct 30 03:39:42.582980 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:42.582640    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" event={"ID":"fd750cb4-4954-4cc5-929a-d40be50a33ac","Type":"ContainerDied","Data":"7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44"}
Oct 30 03:39:45.185075 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:45.185033    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.73:8643/healthz\": dial tcp 10.132.0.73:8643: connect: connection refused"
Oct 30 03:39:46.395151 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.395103    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.74:8643/healthz\": dial tcp 10.132.0.74:8643: connect: connection refused"
Oct 30 03:39:46.426449 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.426425    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:39:46.557258 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.557178    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-cabundle-cert\") pod \"fd750cb4-4954-4cc5-929a-d40be50a33ac\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") "
Oct 30 03:39:46.557258 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.557220    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd750cb4-4954-4cc5-929a-d40be50a33ac-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd750cb4-4954-4cc5-929a-d40be50a33ac-kserve-provision-location\") pod \"fd750cb4-4954-4cc5-929a-d40be50a33ac\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") "
Oct 30 03:39:46.557258 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.557253    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-api-access-brtmz\" (UniqueName: \"kubernetes.io/projected/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-api-access-brtmz\") pod \"fd750cb4-4954-4cc5-929a-d40be50a33ac\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") "
Oct 30 03:39:46.557561 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.557331    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-rbac-proxy-sar-config\") pod \"fd750cb4-4954-4cc5-929a-d40be50a33ac\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") "
Oct 30 03:39:46.557561 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.557366    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd750cb4-4954-4cc5-929a-d40be50a33ac-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd750cb4-4954-4cc5-929a-d40be50a33ac-proxy-tls\") pod \"fd750cb4-4954-4cc5-929a-d40be50a33ac\" (UID: \"fd750cb4-4954-4cc5-929a-d40be50a33ac\") "
Oct 30 03:39:46.557561 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.557492    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd750cb4-4954-4cc5-929a-d40be50a33ac-kserve-provision-location" pod "fd750cb4-4954-4cc5-929a-d40be50a33ac" (UID: "fd750cb4-4954-4cc5-929a-d40be50a33ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:46.557741 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.557579    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd750cb4-4954-4cc5-929a-d40be50a33ac-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:39:46.557741 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.557584    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-cabundle-cert" pod "fd750cb4-4954-4cc5-929a-d40be50a33ac" (UID: "fd750cb4-4954-4cc5-929a-d40be50a33ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:39:46.557819 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.557744    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-rbac-proxy-sar-config" pod "fd750cb4-4954-4cc5-929a-d40be50a33ac" (UID: "fd750cb4-4954-4cc5-929a-d40be50a33ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:39:46.559483 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.559461    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-api-access-brtmz" pod "fd750cb4-4954-4cc5-929a-d40be50a33ac" (UID: "fd750cb4-4954-4cc5-929a-d40be50a33ac"). InnerVolumeSpecName "kube-api-access-brtmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:46.559569 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.559546    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd750cb4-4954-4cc5-929a-d40be50a33ac-proxy-tls" pod "fd750cb4-4954-4cc5-929a-d40be50a33ac" (UID: "fd750cb4-4954-4cc5-929a-d40be50a33ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:39:46.601751 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.601716    4885 generic.go:356] "Generic (PLEG): container finished" podID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerID="a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f" exitCode=0
Oct 30 03:39:46.601961 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.601779    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"
Oct 30 03:39:46.601961 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.601806    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" event={"ID":"fd750cb4-4954-4cc5-929a-d40be50a33ac","Type":"ContainerDied","Data":"a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f"}
Oct 30 03:39:46.601961 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.601911    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k" event={"ID":"fd750cb4-4954-4cc5-929a-d40be50a33ac","Type":"ContainerDied","Data":"e54d07eb1063a287c5f3252d44412b9804274255e4d8cf0ec611abd3b0995bdd"}
Oct 30 03:39:46.601961 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.601950    4885 scope.go:122] "RemoveContainer" containerID="7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44"
Oct 30 03:39:46.610992 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.610974    4885 scope.go:122] "RemoveContainer" containerID="a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f"
Oct 30 03:39:46.618814 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.618798    4885 scope.go:122] "RemoveContainer" containerID="c72bffef4de53d4393a4f3e341eb8a346846a6acfef0e72c0a1c80a597f22e03"
Oct 30 03:39:46.626582 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.626565    4885 scope.go:122] "RemoveContainer" containerID="7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44"
Oct 30 03:39:46.626875 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:39:46.626854    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44\": container with ID starting with 7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44 not found: ID does not exist" containerID="7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44"
Oct 30 03:39:46.626932 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.626882    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44"} err="failed to get container status \"7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44\": rpc error: code = NotFound desc = could not find container \"7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44\": container with ID starting with 7b67695ebd1ba70b89faf80f43c5d946cade6c78d4dd8af99bc20df3d7abee44 not found: ID does not exist"
Oct 30 03:39:46.626932 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.626899    4885 scope.go:122] "RemoveContainer" containerID="a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f"
Oct 30 03:39:46.627226 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:39:46.627210    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f\": container with ID starting with a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f not found: ID does not exist" containerID="a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f"
Oct 30 03:39:46.627278 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.627230    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f"} err="failed to get container status \"a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f\": rpc error: code = NotFound desc = could not find container \"a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f\": container with ID starting with a2b3ef33912e5afa142ce656856da5c30510897fe1b1f7b5dfa4d0a0a04be58f not found: ID does not exist"
Oct 30 03:39:46.627278 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.627242    4885 scope.go:122] "RemoveContainer" containerID="c72bffef4de53d4393a4f3e341eb8a346846a6acfef0e72c0a1c80a597f22e03"
Oct 30 03:39:46.627448 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:39:46.627433    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72bffef4de53d4393a4f3e341eb8a346846a6acfef0e72c0a1c80a597f22e03\": container with ID starting with c72bffef4de53d4393a4f3e341eb8a346846a6acfef0e72c0a1c80a597f22e03 not found: ID does not exist" containerID="c72bffef4de53d4393a4f3e341eb8a346846a6acfef0e72c0a1c80a597f22e03"
Oct 30 03:39:46.627491 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.627452    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72bffef4de53d4393a4f3e341eb8a346846a6acfef0e72c0a1c80a597f22e03"} err="failed to get container status \"c72bffef4de53d4393a4f3e341eb8a346846a6acfef0e72c0a1c80a597f22e03\": rpc error: code = NotFound desc = could not find container \"c72bffef4de53d4393a4f3e341eb8a346846a6acfef0e72c0a1c80a597f22e03\": container with ID starting with c72bffef4de53d4393a4f3e341eb8a346846a6acfef0e72c0a1c80a597f22e03 not found: ID does not exist"
Oct 30 03:39:46.646497 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.646475    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"]
Oct 30 03:39:46.653071 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.653045    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-sq57k"]
Oct 30 03:39:46.658925 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.658907    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brtmz\" (UniqueName: \"kubernetes.io/projected/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-api-access-brtmz\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:39:46.658925 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.658927    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:39:46.659048 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.658938    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd750cb4-4954-4cc5-929a-d40be50a33ac-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:39:46.659048 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:46.658948    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd750cb4-4954-4cc5-929a-d40be50a33ac-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:39:48.624142 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:48.624107    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" path="/var/lib/kubelet/pods/fd750cb4-4954-4cc5-929a-d40be50a33ac/volumes"
Oct 30 03:39:51.395895 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:51.395848    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.74:8643/healthz\": dial tcp 10.132.0.74:8643: connect: connection refused"
Oct 30 03:39:54.887545 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:54.887434    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:39:54.903602 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:54.892411    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:39:54.903602 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:54.892681    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:39:54.903602 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:54.898071    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:39:55.922601 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:39:55.922561    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:39:56.395558 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:56.395518    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.74:8643/healthz\": dial tcp 10.132.0.74:8643: connect: connection refused"
Oct 30 03:39:58.194387 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:58.194342    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"]
Oct 30 03:39:58.195025 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:58.194696    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container" containerID="cri-o://411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd" gracePeriod=30
Oct 30 03:39:58.195115 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:58.195088    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kube-rbac-proxy" containerID="cri-o://60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b" gracePeriod=30
Oct 30 03:39:58.663072 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:58.663039    4885 generic.go:356] "Generic (PLEG): container finished" podID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerID="60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b" exitCode=0
Oct 30 03:39:58.663253 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:39:58.663110    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" event={"ID":"1c7b6b6e-f217-4f26-a720-3d905d6bb261","Type":"ContainerDied","Data":"60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b"}
Oct 30 03:40:00.185764 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.185714    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.72:8643/healthz\": dial tcp 10.132.0.72:8643: connect: connection refused"
Oct 30 03:40:00.191039 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.191012    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 30 03:40:00.675098 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.675067    4885 generic.go:356] "Generic (PLEG): container finished" podID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerID="2dbcbbc630ba60ac9aec1565ef50edc3b9065041a0a64dfba978e2568bf60993" exitCode=137
Oct 30 03:40:00.675282 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.675137    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" event={"ID":"fa95de7c-cf78-4c69-b170-65b79b226eeb","Type":"ContainerDied","Data":"2dbcbbc630ba60ac9aec1565ef50edc3b9065041a0a64dfba978e2568bf60993"}
Oct 30 03:40:00.865581 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.865557    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:40:00.979822 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.979796    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-rbac-proxy-sar-config\") pod \"fa95de7c-cf78-4c69-b170-65b79b226eeb\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") "
Oct 30 03:40:00.979995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.979885    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa95de7c-cf78-4c69-b170-65b79b226eeb-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa95de7c-cf78-4c69-b170-65b79b226eeb-proxy-tls\") pod \"fa95de7c-cf78-4c69-b170-65b79b226eeb\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") "
Oct 30 03:40:00.979995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.979907    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa95de7c-cf78-4c69-b170-65b79b226eeb-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa95de7c-cf78-4c69-b170-65b79b226eeb-kserve-provision-location\") pod \"fa95de7c-cf78-4c69-b170-65b79b226eeb\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") "
Oct 30 03:40:00.979995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.979944    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-cabundle-cert\") pod \"fa95de7c-cf78-4c69-b170-65b79b226eeb\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") "
Oct 30 03:40:00.980114 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.979996    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-api-access-xbsq4\" (UniqueName: \"kubernetes.io/projected/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-api-access-xbsq4\") pod \"fa95de7c-cf78-4c69-b170-65b79b226eeb\" (UID: \"fa95de7c-cf78-4c69-b170-65b79b226eeb\") "
Oct 30 03:40:00.980306 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.980191    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-rbac-proxy-sar-config" pod "fa95de7c-cf78-4c69-b170-65b79b226eeb" (UID: "fa95de7c-cf78-4c69-b170-65b79b226eeb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:00.980392 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.980331    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-cabundle-cert" pod "fa95de7c-cf78-4c69-b170-65b79b226eeb" (UID: "fa95de7c-cf78-4c69-b170-65b79b226eeb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:00.982226 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.982204    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa95de7c-cf78-4c69-b170-65b79b226eeb-proxy-tls" pod "fa95de7c-cf78-4c69-b170-65b79b226eeb" (UID: "fa95de7c-cf78-4c69-b170-65b79b226eeb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:00.982341 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.982330    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-api-access-xbsq4" pod "fa95de7c-cf78-4c69-b170-65b79b226eeb" (UID: "fa95de7c-cf78-4c69-b170-65b79b226eeb"). InnerVolumeSpecName "kube-api-access-xbsq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:00.990478 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:00.990454    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa95de7c-cf78-4c69-b170-65b79b226eeb-kserve-provision-location" pod "fa95de7c-cf78-4c69-b170-65b79b226eeb" (UID: "fa95de7c-cf78-4c69-b170-65b79b226eeb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:01.080628 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:01.080591    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:40:01.080789 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:01.080646    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbsq4\" (UniqueName: \"kubernetes.io/projected/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-api-access-xbsq4\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:40:01.080789 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:01.080657    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa95de7c-cf78-4c69-b170-65b79b226eeb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:40:01.080789 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:01.080667    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa95de7c-cf78-4c69-b170-65b79b226eeb-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:40:01.080789 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:01.080676    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa95de7c-cf78-4c69-b170-65b79b226eeb-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:40:01.681648 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:01.681591    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf" event={"ID":"fa95de7c-cf78-4c69-b170-65b79b226eeb","Type":"ContainerDied","Data":"778aed06f4cebab2ac709f71ecb9efa0887819dc3defd38472bc30230b9655d9"}
Oct 30 03:40:01.681648 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:01.681643    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"
Oct 30 03:40:01.682101 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:01.681690    4885 scope.go:122] "RemoveContainer" containerID="12dfae05266722f7b3b956fda611dcc22ee34e0850199e94e2733bc70f321f4a"
Oct 30 03:40:01.690748 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:01.690729    4885 scope.go:122] "RemoveContainer" containerID="2dbcbbc630ba60ac9aec1565ef50edc3b9065041a0a64dfba978e2568bf60993"
Oct 30 03:40:01.698007 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:01.697989    4885 scope.go:122] "RemoveContainer" containerID="a643f9e73530a9e7928a587d8b6a50cbe3510d0411797cb724bbdde9a2e3dc3e"
Oct 30 03:40:01.736828 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:01.736800    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"]
Oct 30 03:40:01.746770 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:01.746749    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-tensorflow-predictor-649b8d645-2swdf"]
Oct 30 03:40:02.624446 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:02.624416    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" path="/var/lib/kubelet/pods/fa95de7c-cf78-4c69-b170-65b79b226eeb/volumes"
Oct 30 03:40:03.146263 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.146240    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:40:03.296666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.296606    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-rbac-proxy-sar-config\") pod \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") "
Oct 30 03:40:03.296666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.296670    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kserve-provision-location\") pod \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") "
Oct 30 03:40:03.296908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.296717    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-api-access-5xp5f\" (UniqueName: \"kubernetes.io/projected/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-api-access-5xp5f\") pod \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") "
Oct 30 03:40:03.296908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.296754    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-cabundle-cert\") pod \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") "
Oct 30 03:40:03.296908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.296784    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c7b6b6e-f217-4f26-a720-3d905d6bb261-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c7b6b6e-f217-4f26-a720-3d905d6bb261-proxy-tls\") pod \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\" (UID: \"1c7b6b6e-f217-4f26-a720-3d905d6bb261\") "
Oct 30 03:40:03.297082 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.297013    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-rbac-proxy-sar-config" pod "1c7b6b6e-f217-4f26-a720-3d905d6bb261" (UID: "1c7b6b6e-f217-4f26-a720-3d905d6bb261"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:03.297082 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.297049    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kserve-provision-location" pod "1c7b6b6e-f217-4f26-a720-3d905d6bb261" (UID: "1c7b6b6e-f217-4f26-a720-3d905d6bb261"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:03.297190 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.297113    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-cabundle-cert" pod "1c7b6b6e-f217-4f26-a720-3d905d6bb261" (UID: "1c7b6b6e-f217-4f26-a720-3d905d6bb261"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:03.299079 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.299055    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-api-access-5xp5f" pod "1c7b6b6e-f217-4f26-a720-3d905d6bb261" (UID: "1c7b6b6e-f217-4f26-a720-3d905d6bb261"). InnerVolumeSpecName "kube-api-access-5xp5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:03.299164 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.299076    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c7b6b6e-f217-4f26-a720-3d905d6bb261-proxy-tls" pod "1c7b6b6e-f217-4f26-a720-3d905d6bb261" (UID: "1c7b6b6e-f217-4f26-a720-3d905d6bb261"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:03.397502 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.397476    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:40:03.397502 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.397500    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:40:03.397693 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.397509    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xp5f\" (UniqueName: \"kubernetes.io/projected/1c7b6b6e-f217-4f26-a720-3d905d6bb261-kube-api-access-5xp5f\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:40:03.397693 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.397518    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c7b6b6e-f217-4f26-a720-3d905d6bb261-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:40:03.397693 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.397527    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c7b6b6e-f217-4f26-a720-3d905d6bb261-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:40:03.692112 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.692033    4885 generic.go:356] "Generic (PLEG): container finished" podID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerID="411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd" exitCode=0
Oct 30 03:40:03.692112 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.692095    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"
Oct 30 03:40:03.692304 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.692110    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" event={"ID":"1c7b6b6e-f217-4f26-a720-3d905d6bb261","Type":"ContainerDied","Data":"411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd"}
Oct 30 03:40:03.692304 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.692162    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg" event={"ID":"1c7b6b6e-f217-4f26-a720-3d905d6bb261","Type":"ContainerDied","Data":"348d05279cc0715420bea665a6392e819b658d2dba6fcad2db4e93ca0003ee15"}
Oct 30 03:40:03.692304 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.692190    4885 scope.go:122] "RemoveContainer" containerID="60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b"
Oct 30 03:40:03.701453 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.701434    4885 scope.go:122] "RemoveContainer" containerID="411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd"
Oct 30 03:40:03.709310 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.709290    4885 scope.go:122] "RemoveContainer" containerID="94fdb1a8aad60fa80753af160a0c9871c81af2f99432eb65f3a5983b409424cb"
Oct 30 03:40:03.716588 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.716573    4885 scope.go:122] "RemoveContainer" containerID="60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b"
Oct 30 03:40:03.716870 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:40:03.716843    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b\": container with ID starting with 60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b not found: ID does not exist" containerID="60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b"
Oct 30 03:40:03.716933 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.716879    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b"} err="failed to get container status \"60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b\": rpc error: code = NotFound desc = could not find container \"60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b\": container with ID starting with 60cb2a5ec3830de331e538733046aa1c2643d67628976d4f88f28c15fa05ea6b not found: ID does not exist"
Oct 30 03:40:03.716933 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.716897    4885 scope.go:122] "RemoveContainer" containerID="411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd"
Oct 30 03:40:03.717143 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:40:03.717126    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd\": container with ID starting with 411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd not found: ID does not exist" containerID="411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd"
Oct 30 03:40:03.717188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.717148    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd"} err="failed to get container status \"411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd\": rpc error: code = NotFound desc = could not find container \"411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd\": container with ID starting with 411c760bfb43b00bd62f14290660e10132abe0bcbde9b234e33c16bc0d85e2bd not found: ID does not exist"
Oct 30 03:40:03.717188 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.717163    4885 scope.go:122] "RemoveContainer" containerID="94fdb1a8aad60fa80753af160a0c9871c81af2f99432eb65f3a5983b409424cb"
Oct 30 03:40:03.717388 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:40:03.717373    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94fdb1a8aad60fa80753af160a0c9871c81af2f99432eb65f3a5983b409424cb\": container with ID starting with 94fdb1a8aad60fa80753af160a0c9871c81af2f99432eb65f3a5983b409424cb not found: ID does not exist" containerID="94fdb1a8aad60fa80753af160a0c9871c81af2f99432eb65f3a5983b409424cb"
Oct 30 03:40:03.717440 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.717391    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94fdb1a8aad60fa80753af160a0c9871c81af2f99432eb65f3a5983b409424cb"} err="failed to get container status \"94fdb1a8aad60fa80753af160a0c9871c81af2f99432eb65f3a5983b409424cb\": rpc error: code = NotFound desc = could not find container \"94fdb1a8aad60fa80753af160a0c9871c81af2f99432eb65f3a5983b409424cb\": container with ID starting with 94fdb1a8aad60fa80753af160a0c9871c81af2f99432eb65f3a5983b409424cb not found: ID does not exist"
Oct 30 03:40:03.751405 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.751378    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"]
Oct 30 03:40:03.761667 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:03.761649    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-v2-predictor-7c47557b88-fvqbg"]
Oct 30 03:40:04.624207 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:04.624172    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" path="/var/lib/kubelet/pods/1c7b6b6e-f217-4f26-a720-3d905d6bb261/volumes"
Oct 30 03:40:05.915696 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.915665    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"]
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.915975    4885 cpu_manager.go:414] "containerMap: removing container" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kserve-container"
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.915986    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kserve-container"
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.915998    4885 cpu_manager.go:414] "containerMap: removing container" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="storage-initializer"
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916003    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="storage-initializer"
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916011    4885 cpu_manager.go:414] "containerMap: removing container" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container"
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916016    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container"
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916025    4885 cpu_manager.go:414] "containerMap: removing container" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container"
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916029    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container"
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916035    4885 cpu_manager.go:414] "containerMap: removing container" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kube-rbac-proxy"
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916041    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kube-rbac-proxy"
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916050    4885 cpu_manager.go:414] "containerMap: removing container" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kube-rbac-proxy"
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916054    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kube-rbac-proxy"
Oct 30 03:40:05.916064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916068    4885 cpu_manager.go:414] "containerMap: removing container" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kube-rbac-proxy"
Oct 30 03:40:05.916471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916073    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kube-rbac-proxy"
Oct 30 03:40:05.916471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916081    4885 cpu_manager.go:414] "containerMap: removing container" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="storage-initializer"
Oct 30 03:40:05.916471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916085    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="storage-initializer"
Oct 30 03:40:05.916471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916092    4885 cpu_manager.go:414] "containerMap: removing container" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="storage-initializer"
Oct 30 03:40:05.916471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916096    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="storage-initializer"
Oct 30 03:40:05.916471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916141    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kube-rbac-proxy"
Oct 30 03:40:05.916471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916152    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kube-rbac-proxy"
Oct 30 03:40:05.916471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916162    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa95de7c-cf78-4c69-b170-65b79b226eeb" containerName="kserve-container"
Oct 30 03:40:05.916471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916171    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7b6b6e-f217-4f26-a720-3d905d6bb261" containerName="kserve-container"
Oct 30 03:40:05.916471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916180    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kserve-container"
Oct 30 03:40:05.916471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.916189    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd750cb4-4954-4cc5-929a-d40be50a33ac" containerName="kube-rbac-proxy"
Oct 30 03:40:05.921142 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.921117    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:05.945378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.945349    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:40:05.945378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.945368    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:40:05.945573 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.945373    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:40:05.945573 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.945349    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Oct 30 03:40:05.945573 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.945391    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Oct 30 03:40:05.945573 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.945404    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:40:05.945841 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.945818    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:40:05.947633 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:05.947598    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"]
Oct 30 03:40:06.016124 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.016095    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.016264 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.016135    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfg97\" (UniqueName: \"kubernetes.io/projected/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-api-access-tfg97\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.016264 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.016154    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03350354-91b9-47bd-973b-d94ac7deb2fc-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.016264 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.016217    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.016370 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.016269    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03350354-91b9-47bd-973b-d94ac7deb2fc-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.069264 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.069225    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"]
Oct 30 03:40:06.072926 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.072905    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.085389 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.085372    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-promote-predictor-serving-cert\""
Oct 30 03:40:06.095379 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.095357    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"]
Oct 30 03:40:06.116841 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.116814    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.116969 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.116853    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfg97\" (UniqueName: \"kubernetes.io/projected/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-api-access-tfg97\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.116969 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.116878    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03350354-91b9-47bd-973b-d94ac7deb2fc-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.116969 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.116904    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.116969 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.116943    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03350354-91b9-47bd-973b-d94ac7deb2fc-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.117344 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.117326    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03350354-91b9-47bd-973b-d94ac7deb2fc-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.117581 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.117561    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.117649 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.117573    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.119501 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.119476    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03350354-91b9-47bd-973b-d94ac7deb2fc-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.138595 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.138566    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfg97\" (UniqueName: \"kubernetes.io/projected/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-api-access-tfg97\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.217890 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.217859    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-cabundle-cert\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.218096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.217920    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.218096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.218002    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-proxy-tls\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.218096 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.218073    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.218252 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.218109    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsjw5\" (UniqueName: \"kubernetes.io/projected/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-api-access-lsjw5\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.238059 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.238024    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:06.319154 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.319119    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-cabundle-cert\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.319321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.319162    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.319321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.319183    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-proxy-tls\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.319321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.319213    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.319321 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.319244    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsjw5\" (UniqueName: \"kubernetes.io/projected/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-api-access-lsjw5\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.319684 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.319655    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.319898 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.319875    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-cabundle-cert\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.319998 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.319973    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.321931 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.321910    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-proxy-tls\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.340220 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.340194    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsjw5\" (UniqueName: \"kubernetes.io/projected/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-api-access-lsjw5\") pod \"isvc-canary-promote-predictor-5894dcd7f5-5s9d6\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.390094 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.390068    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:06.550575 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.550539    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"]
Oct 30 03:40:06.553256 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:40:06.553224    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03350354_91b9_47bd_973b_d94ac7deb2fc.slice/crio-27255d6e611dcd492e0f5a8b34cd03fb44f19a1ddf12d5b0c5e6cf2b037e8137 WatchSource:0}: Error finding container 27255d6e611dcd492e0f5a8b34cd03fb44f19a1ddf12d5b0c5e6cf2b037e8137: Status 404 returned error can't find the container with id 27255d6e611dcd492e0f5a8b34cd03fb44f19a1ddf12d5b0c5e6cf2b037e8137
Oct 30 03:40:06.685230 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.685192    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"]
Oct 30 03:40:06.688414 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:40:06.688389    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274b4be4_b2f2_4a4c_9fcf_912683ab1ed4.slice/crio-104ad8dcc865a21eadbb971f1f4eccd4001d4ea70f4e33a7b0b8f2d0752b37f6 WatchSource:0}: Error finding container 104ad8dcc865a21eadbb971f1f4eccd4001d4ea70f4e33a7b0b8f2d0752b37f6: Status 404 returned error can't find the container with id 104ad8dcc865a21eadbb971f1f4eccd4001d4ea70f4e33a7b0b8f2d0752b37f6
Oct 30 03:40:06.712464 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.712437    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" event={"ID":"03350354-91b9-47bd-973b-d94ac7deb2fc","Type":"ContainerStarted","Data":"9dbbeb7daabcff179d6c04bc1060f9e4554a82c1590935c4211c5a8edc29ec29"}
Oct 30 03:40:06.712632 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.712507    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" event={"ID":"03350354-91b9-47bd-973b-d94ac7deb2fc","Type":"ContainerStarted","Data":"27255d6e611dcd492e0f5a8b34cd03fb44f19a1ddf12d5b0c5e6cf2b037e8137"}
Oct 30 03:40:06.713744 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:06.713704    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" event={"ID":"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4","Type":"ContainerStarted","Data":"104ad8dcc865a21eadbb971f1f4eccd4001d4ea70f4e33a7b0b8f2d0752b37f6"}
Oct 30 03:40:07.719211 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:07.719176    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" event={"ID":"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4","Type":"ContainerStarted","Data":"bed0c8ea5ff02b14f4445da7b7e9d7f069cb3a2954a0315972af3fd3de15da62"}
Oct 30 03:40:10.733470 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:10.733443    4885 generic.go:356] "Generic (PLEG): container finished" podID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerID="9dbbeb7daabcff179d6c04bc1060f9e4554a82c1590935c4211c5a8edc29ec29" exitCode=0
Oct 30 03:40:10.733909 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:10.733522    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" event={"ID":"03350354-91b9-47bd-973b-d94ac7deb2fc","Type":"ContainerDied","Data":"9dbbeb7daabcff179d6c04bc1060f9e4554a82c1590935c4211c5a8edc29ec29"}
Oct 30 03:40:10.735125 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:10.735107    4885 generic.go:356] "Generic (PLEG): container finished" podID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerID="bed0c8ea5ff02b14f4445da7b7e9d7f069cb3a2954a0315972af3fd3de15da62" exitCode=0
Oct 30 03:40:10.735208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:10.735143    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" event={"ID":"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4","Type":"ContainerDied","Data":"bed0c8ea5ff02b14f4445da7b7e9d7f069cb3a2954a0315972af3fd3de15da62"}
Oct 30 03:40:11.743280 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:11.743248    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" event={"ID":"03350354-91b9-47bd-973b-d94ac7deb2fc","Type":"ContainerStarted","Data":"842bd175f5c3546911d6febb70b3ba0bbe91c1b6957317167821523fdff66d26"}
Oct 30 03:40:11.743746 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:11.743319    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" event={"ID":"03350354-91b9-47bd-973b-d94ac7deb2fc","Type":"ContainerStarted","Data":"c62b3dac83bb684c9b7d66795a17d24605a3da3e273b88f45c134b6259c2fc68"}
Oct 30 03:40:11.743746 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:11.743531    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:11.745378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:11.745351    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" event={"ID":"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4","Type":"ContainerStarted","Data":"b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9"}
Oct 30 03:40:11.745493 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:11.745402    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" event={"ID":"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4","Type":"ContainerStarted","Data":"3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e"}
Oct 30 03:40:11.745586 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:11.745570    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:11.794181 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:11.794120    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podStartSLOduration=2.919714703 podStartE2EDuration="6.794103185s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.874388482s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:11.793284965 +0000 UTC m=+2717.303453990" watchObservedRunningTime="2026-10-30 03:40:11.794103185 +0000 UTC m=+2717.304272214"
Oct 30 03:40:11.833538 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:11.833479    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podStartSLOduration=2.250360123 podStartE2EDuration="5.833461499s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.583101376s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:11.832400402 +0000 UTC m=+2717.342569428" watchObservedRunningTime="2026-10-30 03:40:11.833461499 +0000 UTC m=+2717.343630527"
Oct 30 03:40:12.751106 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:12.751061    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:12.751527 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:12.751127    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:12.752302 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:12.752268    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 03:40:12.752436 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:12.752275    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 03:40:13.756145 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:13.756102    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 03:40:13.756551 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:13.756215    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 03:40:18.760513 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:18.760483    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:40:18.761011 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:18.760992    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:40:18.761074 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:18.761050    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 03:40:18.761419 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:18.761397    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 03:40:25.922899 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:40:25.922864    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:40:28.761499 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:28.761458    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 03:40:28.762083 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:28.761457    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 03:40:38.761006 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:38.760927    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 03:40:38.761457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:38.761432    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 03:40:48.761353 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:48.761311    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 03:40:48.761944 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:48.761445    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 03:40:55.921378 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:40:55.921342    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:40:58.761293 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:58.761257    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 03:40:58.761743 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:40:58.761480    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 03:41:08.761668 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:08.761628    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 03:41:08.761668 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:08.761663    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 03:41:17.703787 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.703752    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"]
Oct 30 03:41:17.708829 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.708808    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.724157 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.724125    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:41:17.724300 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.724199    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:41:17.734140 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.734113    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:41:17.734320 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.734170    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-triton-predictor-serving-cert\""
Oct 30 03:41:17.734427 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.734216    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:41:17.734516 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.734377    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Oct 30 03:41:17.734680 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.734660    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:41:17.736856 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.736828    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"]
Oct 30 03:41:17.797565 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.797536    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6d42e91-40ac-4175-8907-f3acffb32612-kserve-provision-location\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.797732 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.797573    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-cabundle-cert\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.797732 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.797605    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.797854 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.797749    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d42e91-40ac-4175-8907-f3acffb32612-proxy-tls\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.797854 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.797786    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5bvp\" (UniqueName: \"kubernetes.io/projected/c6d42e91-40ac-4175-8907-f3acffb32612-kube-api-access-r5bvp\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.898337 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.898299    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.898529 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.898346    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d42e91-40ac-4175-8907-f3acffb32612-proxy-tls\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.898529 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.898366    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5bvp\" (UniqueName: \"kubernetes.io/projected/c6d42e91-40ac-4175-8907-f3acffb32612-kube-api-access-r5bvp\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.898529 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.898439    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6d42e91-40ac-4175-8907-f3acffb32612-kserve-provision-location\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.898529 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.898468    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-cabundle-cert\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.898956 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.898921    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6d42e91-40ac-4175-8907-f3acffb32612-kserve-provision-location\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.899070 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.899037    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.899121 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.899101    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-cabundle-cert\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.901157 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.901128    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d42e91-40ac-4175-8907-f3acffb32612-proxy-tls\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:17.922528 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:17.922502    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5bvp\" (UniqueName: \"kubernetes.io/projected/c6d42e91-40ac-4175-8907-f3acffb32612-kube-api-access-r5bvp\") pod \"isvc-triton-predictor-7b89746746-rdqqk\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") " pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:18.028022 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:18.027992    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:41:18.339138 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:18.339107    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"]
Oct 30 03:41:18.342567 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:41:18.342539    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d42e91_40ac_4175_8907_f3acffb32612.slice/crio-d2ca7655dcd357221db0443b5e4de48f95e87f2d29a133a5898fef0f9b5ca882 WatchSource:0}: Error finding container d2ca7655dcd357221db0443b5e4de48f95e87f2d29a133a5898fef0f9b5ca882: Status 404 returned error can't find the container with id d2ca7655dcd357221db0443b5e4de48f95e87f2d29a133a5898fef0f9b5ca882
Oct 30 03:41:18.761531 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:18.761142    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 03:41:18.761531 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:18.761469    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 03:41:19.036292 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:19.036198    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" event={"ID":"c6d42e91-40ac-4175-8907-f3acffb32612","Type":"ContainerStarted","Data":"1d1525c89afc0f776a3665fef6f03b38cda79787246feb92497ae4d15df31ddc"}
Oct 30 03:41:19.036292 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:19.036269    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" event={"ID":"c6d42e91-40ac-4175-8907-f3acffb32612","Type":"ContainerStarted","Data":"d2ca7655dcd357221db0443b5e4de48f95e87f2d29a133a5898fef0f9b5ca882"}
Oct 30 03:41:23.054462 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:23.054427    4885 generic.go:356] "Generic (PLEG): container finished" podID="c6d42e91-40ac-4175-8907-f3acffb32612" containerID="1d1525c89afc0f776a3665fef6f03b38cda79787246feb92497ae4d15df31ddc" exitCode=0
Oct 30 03:41:23.054883 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:23.054495    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" event={"ID":"c6d42e91-40ac-4175-8907-f3acffb32612","Type":"ContainerDied","Data":"1d1525c89afc0f776a3665fef6f03b38cda79787246feb92497ae4d15df31ddc"}
Oct 30 03:41:25.944081 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:41:25.944040    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:41:28.763402 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:28.761000    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 03:41:28.764503 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:28.764116    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:41:29.620968 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:29.620566    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 03:41:39.622154 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:39.622122    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:41:46.062464 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:46.062416    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"]
Oct 30 03:41:46.063109 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:46.062727    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container" containerID="cri-o://3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e" gracePeriod=30
Oct 30 03:41:46.063513 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:46.063486    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kube-rbac-proxy" containerID="cri-o://b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9" gracePeriod=30
Oct 30 03:41:46.219583 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:46.219536    4885 generic.go:356] "Generic (PLEG): container finished" podID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerID="b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9" exitCode=0
Oct 30 03:41:46.219877 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:46.219645    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" event={"ID":"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4","Type":"ContainerDied","Data":"b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9"}
Oct 30 03:41:48.756502 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:48.756463    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.76:8643/healthz\": dial tcp 10.132.0.76:8643: connect: connection refused"
Oct 30 03:41:48.762123 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:48.762089    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 30 03:41:51.892458 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:51.892392    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:41:52.028369 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.028332    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-api-access-lsjw5\" (UniqueName: \"kubernetes.io/projected/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-api-access-lsjw5\") pod \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") "
Oct 30 03:41:52.028549 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.028400    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-rbac-proxy-sar-config\") pod \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") "
Oct 30 03:41:52.028549 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.028454    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-proxy-tls\" (UniqueName: \"kubernetes.io/secret/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-proxy-tls\") pod \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") "
Oct 30 03:41:52.028549 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.028520    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kserve-provision-location\") pod \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") "
Oct 30 03:41:52.028745 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.028603    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-cabundle-cert\") pod \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\" (UID: \"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4\") "
Oct 30 03:41:52.029175 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.029142    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-cabundle-cert" pod "274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" (UID: "274b4be4-b2f2-4a4c-9fcf-912683ab1ed4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:52.029472 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.029440    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-rbac-proxy-sar-config" pod "274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" (UID: "274b4be4-b2f2-4a4c-9fcf-912683ab1ed4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:52.029589 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.029562    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kserve-provision-location" pod "274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" (UID: "274b4be4-b2f2-4a4c-9fcf-912683ab1ed4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:41:52.030821 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.030786    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-api-access-lsjw5" pod "274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" (UID: "274b4be4-b2f2-4a4c-9fcf-912683ab1ed4"). InnerVolumeSpecName "kube-api-access-lsjw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:52.031628 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.031575    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-proxy-tls" pod "274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" (UID: "274b4be4-b2f2-4a4c-9fcf-912683ab1ed4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:52.129475 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.129440    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:41:52.129475 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.129470    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsjw5\" (UniqueName: \"kubernetes.io/projected/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-api-access-lsjw5\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:41:52.129751 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.129483    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:41:52.129751 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.129496    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:41:52.129751 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.129507    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:41:52.259978 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.259932    4885 generic.go:356] "Generic (PLEG): container finished" podID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerID="3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e" exitCode=0
Oct 30 03:41:52.260171 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.260022    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" event={"ID":"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4","Type":"ContainerDied","Data":"3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e"}
Oct 30 03:41:52.260171 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.260082    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6" event={"ID":"274b4be4-b2f2-4a4c-9fcf-912683ab1ed4","Type":"ContainerDied","Data":"104ad8dcc865a21eadbb971f1f4eccd4001d4ea70f4e33a7b0b8f2d0752b37f6"}
Oct 30 03:41:52.260171 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.260121    4885 scope.go:122] "RemoveContainer" containerID="b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9"
Oct 30 03:41:52.260343 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.260323    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"
Oct 30 03:41:52.275796 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.275779    4885 scope.go:122] "RemoveContainer" containerID="3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e"
Oct 30 03:41:52.287491 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.287465    4885 scope.go:122] "RemoveContainer" containerID="bed0c8ea5ff02b14f4445da7b7e9d7f069cb3a2954a0315972af3fd3de15da62"
Oct 30 03:41:52.300545 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.300519    4885 scope.go:122] "RemoveContainer" containerID="b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9"
Oct 30 03:41:52.301057 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:41:52.300932    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9\": container with ID starting with b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9 not found: ID does not exist" containerID="b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9"
Oct 30 03:41:52.301057 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.300972    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9"} err="failed to get container status \"b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9\": rpc error: code = NotFound desc = could not find container \"b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9\": container with ID starting with b6f273529040a1ecc4cefc4b945904f03aa30ce7ae709f2a6262ca4c58e7f5e9 not found: ID does not exist"
Oct 30 03:41:52.301057 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.300991    4885 scope.go:122] "RemoveContainer" containerID="3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e"
Oct 30 03:41:52.301390 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:41:52.301368    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e\": container with ID starting with 3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e not found: ID does not exist" containerID="3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e"
Oct 30 03:41:52.301481 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.301406    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e"} err="failed to get container status \"3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e\": rpc error: code = NotFound desc = could not find container \"3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e\": container with ID starting with 3f6230a125beb4d0472ec443f5516ed24d7646bbc00703306529a992f7b8c55e not found: ID does not exist"
Oct 30 03:41:52.301481 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.301426    4885 scope.go:122] "RemoveContainer" containerID="bed0c8ea5ff02b14f4445da7b7e9d7f069cb3a2954a0315972af3fd3de15da62"
Oct 30 03:41:52.301812 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:41:52.301780    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed0c8ea5ff02b14f4445da7b7e9d7f069cb3a2954a0315972af3fd3de15da62\": container with ID starting with bed0c8ea5ff02b14f4445da7b7e9d7f069cb3a2954a0315972af3fd3de15da62 not found: ID does not exist" containerID="bed0c8ea5ff02b14f4445da7b7e9d7f069cb3a2954a0315972af3fd3de15da62"
Oct 30 03:41:52.301935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.301813    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed0c8ea5ff02b14f4445da7b7e9d7f069cb3a2954a0315972af3fd3de15da62"} err="failed to get container status \"bed0c8ea5ff02b14f4445da7b7e9d7f069cb3a2954a0315972af3fd3de15da62\": rpc error: code = NotFound desc = could not find container \"bed0c8ea5ff02b14f4445da7b7e9d7f069cb3a2954a0315972af3fd3de15da62\": container with ID starting with bed0c8ea5ff02b14f4445da7b7e9d7f069cb3a2954a0315972af3fd3de15da62 not found: ID does not exist"
Oct 30 03:41:52.307137 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.307109    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"]
Oct 30 03:41:52.311101 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.311079    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-5s9d6"]
Oct 30 03:41:52.625457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:41:52.625376    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" path="/var/lib/kubelet/pods/274b4be4-b2f2-4a4c-9fcf-912683ab1ed4/volumes"
Oct 30 03:41:55.936179 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:41:55.936142    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:42:25.984051 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:42:25.984010    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:42:55.925029 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:42:55.924989    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:43:20.702751 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:20.702713    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" event={"ID":"c6d42e91-40ac-4175-8907-f3acffb32612","Type":"ContainerStarted","Data":"85f7d236b44100167de516206dcb0c3288107a59873193ccaed2872ab699c8d9"}
Oct 30 03:43:20.703220 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:20.702787    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" event={"ID":"c6d42e91-40ac-4175-8907-f3acffb32612","Type":"ContainerStarted","Data":"f9680f5fd74e6cfa910d41fa3632989a0acd724028a09e4dde4d9bf4eaa14fc3"}
Oct 30 03:43:20.703220 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:20.702902    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:43:20.756442 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:20.756394    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" podStartSLOduration=3.189189683 podStartE2EDuration="2m3.756380319s" totalImagesPullingTime="1m56.817227047s" totalInitContainerRuntime="3.749963589s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:20.75540783 +0000 UTC m=+2906.265576880" watchObservedRunningTime="2026-10-30 03:43:20.756380319 +0000 UTC m=+2906.266549392"
Oct 30 03:43:21.707003 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:21.706972    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:43:21.708232 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:21.708208    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 30 03:43:22.710442 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:22.710405    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 30 03:43:25.927687 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:43:25.927654    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:43:26.736663 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:26.736625    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"]
Oct 30 03:43:26.736976 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:26.736884    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" containerID="cri-o://c62b3dac83bb684c9b7d66795a17d24605a3da3e273b88f45c134b6259c2fc68" gracePeriod=30
Oct 30 03:43:26.737817 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:26.737710    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kube-rbac-proxy" containerID="cri-o://842bd175f5c3546911d6febb70b3ba0bbe91c1b6957317167821523fdff66d26" gracePeriod=30
Oct 30 03:43:27.715720 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:27.715691    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:43:27.716465 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:27.716446    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:43:27.734629 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:27.734585    4885 generic.go:356] "Generic (PLEG): container finished" podID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerID="842bd175f5c3546911d6febb70b3ba0bbe91c1b6957317167821523fdff66d26" exitCode=0
Oct 30 03:43:27.734806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:27.734636    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" event={"ID":"03350354-91b9-47bd-973b-d94ac7deb2fc","Type":"ContainerDied","Data":"842bd175f5c3546911d6febb70b3ba0bbe91c1b6957317167821523fdff66d26"}
Oct 30 03:43:28.756847 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:28.756809    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.75:8643/healthz\": dial tcp 10.132.0.75:8643: connect: connection refused"
Oct 30 03:43:29.620936 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:29.620892    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 30 03:43:31.755606 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:31.755573    4885 generic.go:356] "Generic (PLEG): container finished" podID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerID="c62b3dac83bb684c9b7d66795a17d24605a3da3e273b88f45c134b6259c2fc68" exitCode=0
Oct 30 03:43:31.756012 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:31.755653    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" event={"ID":"03350354-91b9-47bd-973b-d94ac7deb2fc","Type":"ContainerDied","Data":"c62b3dac83bb684c9b7d66795a17d24605a3da3e273b88f45c134b6259c2fc68"}
Oct 30 03:43:32.153437 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.153407    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:43:32.176387 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.176349    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-rbac-proxy-sar-config\") pod \"03350354-91b9-47bd-973b-d94ac7deb2fc\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") "
Oct 30 03:43:32.176517 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.176410    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03350354-91b9-47bd-973b-d94ac7deb2fc-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03350354-91b9-47bd-973b-d94ac7deb2fc-kserve-provision-location\") pod \"03350354-91b9-47bd-973b-d94ac7deb2fc\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") "
Oct 30 03:43:32.176517 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.176430    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-api-access-tfg97\" (UniqueName: \"kubernetes.io/projected/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-api-access-tfg97\") pod \"03350354-91b9-47bd-973b-d94ac7deb2fc\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") "
Oct 30 03:43:32.176517 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.176489    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03350354-91b9-47bd-973b-d94ac7deb2fc-proxy-tls\" (UniqueName: \"kubernetes.io/secret/03350354-91b9-47bd-973b-d94ac7deb2fc-proxy-tls\") pod \"03350354-91b9-47bd-973b-d94ac7deb2fc\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") "
Oct 30 03:43:32.176708 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.176523    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-cabundle-cert\") pod \"03350354-91b9-47bd-973b-d94ac7deb2fc\" (UID: \"03350354-91b9-47bd-973b-d94ac7deb2fc\") "
Oct 30 03:43:32.176829 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.176724    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-rbac-proxy-sar-config" pod "03350354-91b9-47bd-973b-d94ac7deb2fc" (UID: "03350354-91b9-47bd-973b-d94ac7deb2fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:32.176829 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.176764    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03350354-91b9-47bd-973b-d94ac7deb2fc-kserve-provision-location" pod "03350354-91b9-47bd-973b-d94ac7deb2fc" (UID: "03350354-91b9-47bd-973b-d94ac7deb2fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:32.177050 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.177026    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-cabundle-cert" pod "03350354-91b9-47bd-973b-d94ac7deb2fc" (UID: "03350354-91b9-47bd-973b-d94ac7deb2fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:32.178788 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.178767    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-api-access-tfg97" pod "03350354-91b9-47bd-973b-d94ac7deb2fc" (UID: "03350354-91b9-47bd-973b-d94ac7deb2fc"). InnerVolumeSpecName "kube-api-access-tfg97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:32.178892 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.178870    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03350354-91b9-47bd-973b-d94ac7deb2fc-proxy-tls" pod "03350354-91b9-47bd-973b-d94ac7deb2fc" (UID: "03350354-91b9-47bd-973b-d94ac7deb2fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:32.277860 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.277821    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:43:32.277860 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.277851    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03350354-91b9-47bd-973b-d94ac7deb2fc-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:43:32.277860 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.277861    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfg97\" (UniqueName: \"kubernetes.io/projected/03350354-91b9-47bd-973b-d94ac7deb2fc-kube-api-access-tfg97\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:43:32.277860 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.277872    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03350354-91b9-47bd-973b-d94ac7deb2fc-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:43:32.278126 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.277881    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03350354-91b9-47bd-973b-d94ac7deb2fc-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:43:32.761965 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.761929    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr" event={"ID":"03350354-91b9-47bd-973b-d94ac7deb2fc","Type":"ContainerDied","Data":"27255d6e611dcd492e0f5a8b34cd03fb44f19a1ddf12d5b0c5e6cf2b037e8137"}
Oct 30 03:43:32.762454 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.762003    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"
Oct 30 03:43:32.762454 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.762017    4885 scope.go:122] "RemoveContainer" containerID="842bd175f5c3546911d6febb70b3ba0bbe91c1b6957317167821523fdff66d26"
Oct 30 03:43:32.770658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.770636    4885 scope.go:122] "RemoveContainer" containerID="c62b3dac83bb684c9b7d66795a17d24605a3da3e273b88f45c134b6259c2fc68"
Oct 30 03:43:32.778158 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.778141    4885 scope.go:122] "RemoveContainer" containerID="9dbbeb7daabcff179d6c04bc1060f9e4554a82c1590935c4211c5a8edc29ec29"
Oct 30 03:43:32.807911 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.807873    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"]
Oct 30 03:43:32.817928 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:32.817904    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-nptdr"]
Oct 30 03:43:34.454190 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454150    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"]
Oct 30 03:43:34.454773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454638    4885 cpu_manager.go:414] "containerMap: removing container" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="storage-initializer"
Oct 30 03:43:34.454773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454654    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="storage-initializer"
Oct 30 03:43:34.454773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454669    4885 cpu_manager.go:414] "containerMap: removing container" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container"
Oct 30 03:43:34.454773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454674    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container"
Oct 30 03:43:34.454773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454687    4885 cpu_manager.go:414] "containerMap: removing container" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container"
Oct 30 03:43:34.454773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454693    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container"
Oct 30 03:43:34.454773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454705    4885 cpu_manager.go:414] "containerMap: removing container" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kube-rbac-proxy"
Oct 30 03:43:34.454773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454710    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kube-rbac-proxy"
Oct 30 03:43:34.454773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454721    4885 cpu_manager.go:414] "containerMap: removing container" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="storage-initializer"
Oct 30 03:43:34.454773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454726    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="storage-initializer"
Oct 30 03:43:34.454773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454734    4885 cpu_manager.go:414] "containerMap: removing container" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kube-rbac-proxy"
Oct 30 03:43:34.454773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454738    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kube-rbac-proxy"
Oct 30 03:43:34.455159 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454786    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kserve-container"
Oct 30 03:43:34.455159 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454794    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kserve-container"
Oct 30 03:43:34.455159 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454801    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="274b4be4-b2f2-4a4c-9fcf-912683ab1ed4" containerName="kube-rbac-proxy"
Oct 30 03:43:34.455159 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.454808    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" containerName="kube-rbac-proxy"
Oct 30 03:43:34.489592 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.489562    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"]
Oct 30 03:43:34.489789 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.489706    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.558695 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.558663    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:43:34.558695 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.558690    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:43:34.558923 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.558757    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 30 03:43:34.561480 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.561457    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Oct 30 03:43:34.561544 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.561477    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:43:34.561544 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.561457    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:43:34.561544 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.561517    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:43:34.598122 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.598087    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-proxy-tls\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.598297 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.598128    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.598297 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.598255    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.598369 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.598301    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4dcd\" (UniqueName: \"kubernetes.io/projected/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-api-access-q4dcd\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.598369 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.598329    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.624395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.624364    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03350354-91b9-47bd-973b-d94ac7deb2fc" path="/var/lib/kubelet/pods/03350354-91b9-47bd-973b-d94ac7deb2fc/volumes"
Oct 30 03:43:34.698825 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.698783    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4dcd\" (UniqueName: \"kubernetes.io/projected/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-api-access-q4dcd\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.699036 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.698978    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.699110 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.699085    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-proxy-tls\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.699183 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.699161    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.699248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.699229    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.699636 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.699586    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.699806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.699783    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.699903 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.699888    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.701787 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.701753    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-proxy-tls\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.720541 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.720479    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4dcd\" (UniqueName: \"kubernetes.io/projected/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-api-access-q4dcd\") pod \"isvc-canary-rollback-predictor-5769c8765-zl6kf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:34.865883 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:34.865844    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:35.220261 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:35.220213    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"]
Oct 30 03:43:35.223487 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:43:35.223456    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8487d62d_b1fb_435f_a00f_2b61bcfbf7cf.slice/crio-850a8cd77b720e9e907231609e118dddb78148196962e09af1a18cd07751528e WatchSource:0}: Error finding container 850a8cd77b720e9e907231609e118dddb78148196962e09af1a18cd07751528e: Status 404 returned error can't find the container with id 850a8cd77b720e9e907231609e118dddb78148196962e09af1a18cd07751528e
Oct 30 03:43:35.777997 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:35.777954    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" event={"ID":"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf","Type":"ContainerStarted","Data":"3a65fca9322b9e7260bb2d09e641a8e86e39bb6b48e7b5ee07772a9490156678"}
Oct 30 03:43:35.778382 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:35.778033    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" event={"ID":"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf","Type":"ContainerStarted","Data":"850a8cd77b720e9e907231609e118dddb78148196962e09af1a18cd07751528e"}
Oct 30 03:43:39.795584 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:39.795556    4885 generic.go:356] "Generic (PLEG): container finished" podID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerID="3a65fca9322b9e7260bb2d09e641a8e86e39bb6b48e7b5ee07772a9490156678" exitCode=0
Oct 30 03:43:39.796087 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:39.795651    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" event={"ID":"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf","Type":"ContainerDied","Data":"3a65fca9322b9e7260bb2d09e641a8e86e39bb6b48e7b5ee07772a9490156678"}
Oct 30 03:43:40.241946 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:40.241918    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"]
Oct 30 03:43:40.242212 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:40.242183    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="kserve-container" containerID="cri-o://f9680f5fd74e6cfa910d41fa3632989a0acd724028a09e4dde4d9bf4eaa14fc3" gracePeriod=30
Oct 30 03:43:40.242330 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:40.242265    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="kube-rbac-proxy" containerID="cri-o://85f7d236b44100167de516206dcb0c3288107a59873193ccaed2872ab699c8d9" gracePeriod=30
Oct 30 03:43:40.801973 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:40.801923    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" event={"ID":"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf","Type":"ContainerStarted","Data":"861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63"}
Oct 30 03:43:40.802400 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:40.802000    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" event={"ID":"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf","Type":"ContainerStarted","Data":"3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697"}
Oct 30 03:43:40.802400 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:40.802203    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:40.804029 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:40.804005    4885 generic.go:356] "Generic (PLEG): container finished" podID="c6d42e91-40ac-4175-8907-f3acffb32612" containerID="85f7d236b44100167de516206dcb0c3288107a59873193ccaed2872ab699c8d9" exitCode=0
Oct 30 03:43:40.804149 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:40.804074    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" event={"ID":"c6d42e91-40ac-4175-8907-f3acffb32612","Type":"ContainerDied","Data":"85f7d236b44100167de516206dcb0c3288107a59873193ccaed2872ab699c8d9"}
Oct 30 03:43:40.843014 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:40.842970    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podStartSLOduration=2.866260612 podStartE2EDuration="6.842957871s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.976697259s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:40.841881277 +0000 UTC m=+2926.352050321" watchObservedRunningTime="2026-10-30 03:43:40.842957871 +0000 UTC m=+2926.353126897"
Oct 30 03:43:41.808381 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:41.808351    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:41.809536 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:41.809510    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 03:43:42.711090 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:42.711051    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.77:8643/healthz\": dial tcp 10.132.0.77:8643: connect: connection refused"
Oct 30 03:43:42.814785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:42.814706    4885 generic.go:356] "Generic (PLEG): container finished" podID="c6d42e91-40ac-4175-8907-f3acffb32612" containerID="f9680f5fd74e6cfa910d41fa3632989a0acd724028a09e4dde4d9bf4eaa14fc3" exitCode=0
Oct 30 03:43:42.814785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:42.814776    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" event={"ID":"c6d42e91-40ac-4175-8907-f3acffb32612","Type":"ContainerDied","Data":"f9680f5fd74e6cfa910d41fa3632989a0acd724028a09e4dde4d9bf4eaa14fc3"}
Oct 30 03:43:42.815304 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:42.815047    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 03:43:43.213193 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.213165    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:43:43.267731 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.267698    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-cabundle-cert\") pod \"c6d42e91-40ac-4175-8907-f3acffb32612\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") "
Oct 30 03:43:43.267919 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.267802    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d42e91-40ac-4175-8907-f3acffb32612-kube-api-access-r5bvp\" (UniqueName: \"kubernetes.io/projected/c6d42e91-40ac-4175-8907-f3acffb32612-kube-api-access-r5bvp\") pod \"c6d42e91-40ac-4175-8907-f3acffb32612\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") "
Oct 30 03:43:43.267919 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.267834    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-kube-rbac-proxy-sar-config\") pod \"c6d42e91-40ac-4175-8907-f3acffb32612\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") "
Oct 30 03:43:43.267919 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.267873    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d42e91-40ac-4175-8907-f3acffb32612-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d42e91-40ac-4175-8907-f3acffb32612-proxy-tls\") pod \"c6d42e91-40ac-4175-8907-f3acffb32612\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") "
Oct 30 03:43:43.267919 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.267912    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d42e91-40ac-4175-8907-f3acffb32612-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6d42e91-40ac-4175-8907-f3acffb32612-kserve-provision-location\") pod \"c6d42e91-40ac-4175-8907-f3acffb32612\" (UID: \"c6d42e91-40ac-4175-8907-f3acffb32612\") "
Oct 30 03:43:43.268320 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.268282    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-cabundle-cert" pod "c6d42e91-40ac-4175-8907-f3acffb32612" (UID: "c6d42e91-40ac-4175-8907-f3acffb32612"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:43.268822 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.268729    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:43:43.268822 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.268754    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d42e91-40ac-4175-8907-f3acffb32612-kserve-provision-location" pod "c6d42e91-40ac-4175-8907-f3acffb32612" (UID: "c6d42e91-40ac-4175-8907-f3acffb32612"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:43.268822 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.268764    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-kube-rbac-proxy-sar-config" pod "c6d42e91-40ac-4175-8907-f3acffb32612" (UID: "c6d42e91-40ac-4175-8907-f3acffb32612"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:43.270672 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.270639    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d42e91-40ac-4175-8907-f3acffb32612-proxy-tls" pod "c6d42e91-40ac-4175-8907-f3acffb32612" (UID: "c6d42e91-40ac-4175-8907-f3acffb32612"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:43.270769 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.270752    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d42e91-40ac-4175-8907-f3acffb32612-kube-api-access-r5bvp" pod "c6d42e91-40ac-4175-8907-f3acffb32612" (UID: "c6d42e91-40ac-4175-8907-f3acffb32612"). InnerVolumeSpecName "kube-api-access-r5bvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:43.369547 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.369470    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5bvp\" (UniqueName: \"kubernetes.io/projected/c6d42e91-40ac-4175-8907-f3acffb32612-kube-api-access-r5bvp\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:43:43.369547 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.369501    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6d42e91-40ac-4175-8907-f3acffb32612-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:43:43.369547 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.369514    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d42e91-40ac-4175-8907-f3acffb32612-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:43:43.369547 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.369527    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6d42e91-40ac-4175-8907-f3acffb32612-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:43:43.822031 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.821997    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk" event={"ID":"c6d42e91-40ac-4175-8907-f3acffb32612","Type":"ContainerDied","Data":"d2ca7655dcd357221db0443b5e4de48f95e87f2d29a133a5898fef0f9b5ca882"}
Oct 30 03:43:43.822451 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.822063    4885 scope.go:122] "RemoveContainer" containerID="85f7d236b44100167de516206dcb0c3288107a59873193ccaed2872ab699c8d9"
Oct 30 03:43:43.822451 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.822070    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"
Oct 30 03:43:43.831137 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.831117    4885 scope.go:122] "RemoveContainer" containerID="f9680f5fd74e6cfa910d41fa3632989a0acd724028a09e4dde4d9bf4eaa14fc3"
Oct 30 03:43:43.839047 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.839030    4885 scope.go:122] "RemoveContainer" containerID="1d1525c89afc0f776a3665fef6f03b38cda79787246feb92497ae4d15df31ddc"
Oct 30 03:43:43.878153 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.878129    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"]
Oct 30 03:43:43.881793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:43.881775    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-triton-predictor-7b89746746-rdqqk"]
Oct 30 03:43:44.623376 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:44.623340    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" path="/var/lib/kubelet/pods/c6d42e91-40ac-4175-8907-f3acffb32612/volumes"
Oct 30 03:43:45.831519 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.831482    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"]
Oct 30 03:43:45.832168 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.831996    4885 cpu_manager.go:414] "containerMap: removing container" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="kserve-container"
Oct 30 03:43:45.832168 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.832016    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="kserve-container"
Oct 30 03:43:45.832168 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.832034    4885 cpu_manager.go:414] "containerMap: removing container" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="storage-initializer"
Oct 30 03:43:45.832168 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.832042    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="storage-initializer"
Oct 30 03:43:45.832168 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.832067    4885 cpu_manager.go:414] "containerMap: removing container" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="kube-rbac-proxy"
Oct 30 03:43:45.832168 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.832074    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="kube-rbac-proxy"
Oct 30 03:43:45.832168 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.832163    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="kserve-container"
Oct 30 03:43:45.832572 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.832179    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d42e91-40ac-4175-8907-f3acffb32612" containerName="kube-rbac-proxy"
Oct 30 03:43:45.863839 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.863813    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"]
Oct 30 03:43:45.863981 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.863947    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.878745 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.878721    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:43:45.878865 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.878817    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Oct 30 03:43:45.889147 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.889123    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:43:45.889521 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.889497    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-xgboost-predictor-serving-cert\""
Oct 30 03:43:45.889678 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.889123    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:43:45.889816 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.889230    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:43:45.889901 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.889745    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:43:45.895195 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.895168    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.895303 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.895275    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-cabundle-cert\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.895362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.895317    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kserve-provision-location\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.895406 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.895382    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq6m6\" (UniqueName: \"kubernetes.io/projected/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-api-access-sq6m6\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.895483 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.895463    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-proxy-tls\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.996675 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.996636    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-cabundle-cert\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.996675 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.996678    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kserve-provision-location\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.996941 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.996701    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq6m6\" (UniqueName: \"kubernetes.io/projected/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-api-access-sq6m6\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.996941 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.996730    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-proxy-tls\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.996941 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.996768    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.997201 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.997171    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kserve-provision-location\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.997419 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.997395    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-cabundle-cert\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.997480 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.997453    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:45.999517 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:45.999493    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-proxy-tls\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:46.018246 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:46.018221    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq6m6\" (UniqueName: \"kubernetes.io/projected/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-api-access-sq6m6\") pod \"isvc-xgboost-predictor-7b554dbc6c-84mqf\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:46.238656 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:46.238604    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:43:46.563092 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:46.563001    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"]
Oct 30 03:43:46.566463 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:43:46.566437    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24b5da3_18f5_4b1a_afb3_fd10a7bea3bd.slice/crio-01f4cef2a211e7190462dfb2ffac038372866148aaa77e57ff016abf427a8278 WatchSource:0}: Error finding container 01f4cef2a211e7190462dfb2ffac038372866148aaa77e57ff016abf427a8278: Status 404 returned error can't find the container with id 01f4cef2a211e7190462dfb2ffac038372866148aaa77e57ff016abf427a8278
Oct 30 03:43:46.838177 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:46.838094    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" event={"ID":"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd","Type":"ContainerStarted","Data":"05ccc1dcad159222c08c9e948f010b795e8988514ba0c14c24657eea43049297"}
Oct 30 03:43:46.838177 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:46.838158    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" event={"ID":"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd","Type":"ContainerStarted","Data":"01f4cef2a211e7190462dfb2ffac038372866148aaa77e57ff016abf427a8278"}
Oct 30 03:43:47.819399 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:47.819374    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:43:47.819955 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:47.819928    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 03:43:50.857193 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:50.857116    4885 generic.go:356] "Generic (PLEG): container finished" podID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerID="05ccc1dcad159222c08c9e948f010b795e8988514ba0c14c24657eea43049297" exitCode=0
Oct 30 03:43:50.857546 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:50.857188    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" event={"ID":"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd","Type":"ContainerDied","Data":"05ccc1dcad159222c08c9e948f010b795e8988514ba0c14c24657eea43049297"}
Oct 30 03:43:55.926150 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:43:55.926108    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:43:57.819986 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:43:57.819952    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 03:44:07.820630 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:07.820574    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 03:44:09.954984 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:09.954947    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" event={"ID":"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd","Type":"ContainerStarted","Data":"6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239"}
Oct 30 03:44:09.955395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:09.955013    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" event={"ID":"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd","Type":"ContainerStarted","Data":"9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1"}
Oct 30 03:44:09.955395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:09.955357    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:44:09.955511 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:09.955494    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:44:09.956670 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:09.956646    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 03:44:10.007138 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:10.007091    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" podStartSLOduration=3.014874032 podStartE2EDuration="25.007074792s" totalImagesPullingTime="18.15345089s" totalInitContainerRuntime="3.83874987s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:10.006273326 +0000 UTC m=+2955.516442363" watchObservedRunningTime="2026-10-30 03:44:10.007074792 +0000 UTC m=+2955.517243822"
Oct 30 03:44:10.959705 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:10.959665    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 03:44:15.963683 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:15.963655    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:44:15.964172 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:15.964105    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 03:44:17.820127 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:17.820083    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 03:44:25.922131 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:44:25.922094    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:44:25.964958 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:25.964924    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 03:44:27.820225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:27.820187    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 03:44:35.964886 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:35.964846    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 03:44:37.820109 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:37.820072    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 03:44:45.964838 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:45.964798    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 03:44:47.819811 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:47.819774    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 03:44:54.921450 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:54.921316    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:44:54.958034 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:54.925476    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:44:54.958034 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:54.927328    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:44:54.958034 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:54.931350    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:44:55.921290 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:44:55.921250    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:44:55.964425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:55.964391    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 03:44:57.820030 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:44:57.819988    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 30 03:45:05.620427 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:05.620396    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:45:05.965014 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:05.964973    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 30 03:45:06.009352 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:06.009322    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"]
Oct 30 03:45:07.200005 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:07.199958    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container" containerID="cri-o://3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697" gracePeriod=30
Oct 30 03:45:07.200475 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:07.200018    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kube-rbac-proxy" containerID="cri-o://861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63" gracePeriod=30
Oct 30 03:45:07.815837 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:07.815801    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.78:8643/healthz\": dial tcp 10.132.0.78:8643: connect: connection refused"
Oct 30 03:45:08.216091 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:08.216059    4885 generic.go:356] "Generic (PLEG): container finished" podID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerID="861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63" exitCode=0
Oct 30 03:45:08.216522 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:08.216132    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" event={"ID":"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf","Type":"ContainerDied","Data":"861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63"}
Oct 30 03:45:11.940185 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:11.940154    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:45:12.027598 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.027517    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kserve-provision-location\") pod \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") "
Oct 30 03:45:12.027598 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.027584    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-cabundle-cert\") pod \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") "
Oct 30 03:45:12.027846 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.027650    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-api-access-q4dcd\" (UniqueName: \"kubernetes.io/projected/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-api-access-q4dcd\") pod \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") "
Oct 30 03:45:12.027846 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.027675    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-rbac-proxy-sar-config\") pod \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") "
Oct 30 03:45:12.027846 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.027711    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-proxy-tls\") pod \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\" (UID: \"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf\") "
Oct 30 03:45:12.027998 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.027863    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kserve-provision-location" pod "8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" (UID: "8487d62d-b1fb-435f-a00f-2b61bcfbf7cf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:12.027998 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.027969    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-cabundle-cert" pod "8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" (UID: "8487d62d-b1fb-435f-a00f-2b61bcfbf7cf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:12.028071 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.028022    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-rbac-proxy-sar-config" pod "8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" (UID: "8487d62d-b1fb-435f-a00f-2b61bcfbf7cf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:12.029852 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.029827    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-api-access-q4dcd" pod "8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" (UID: "8487d62d-b1fb-435f-a00f-2b61bcfbf7cf"). InnerVolumeSpecName "kube-api-access-q4dcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:12.029955 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.029897    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-proxy-tls" pod "8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" (UID: "8487d62d-b1fb-435f-a00f-2b61bcfbf7cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:12.128582 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.128556    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:45:12.128582 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.128579    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:45:12.128753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.128589    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4dcd\" (UniqueName: \"kubernetes.io/projected/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-api-access-q4dcd\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:45:12.128753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.128598    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:45:12.128753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.128607    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:45:12.235345 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.235313    4885 generic.go:356] "Generic (PLEG): container finished" podID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerID="3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697" exitCode=0
Oct 30 03:45:12.235531 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.235376    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"
Oct 30 03:45:12.235531 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.235377    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" event={"ID":"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf","Type":"ContainerDied","Data":"3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697"}
Oct 30 03:45:12.235531 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.235511    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf" event={"ID":"8487d62d-b1fb-435f-a00f-2b61bcfbf7cf","Type":"ContainerDied","Data":"850a8cd77b720e9e907231609e118dddb78148196962e09af1a18cd07751528e"}
Oct 30 03:45:12.235716 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.235547    4885 scope.go:122] "RemoveContainer" containerID="861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63"
Oct 30 03:45:12.244809 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.244791    4885 scope.go:122] "RemoveContainer" containerID="3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697"
Oct 30 03:45:12.252558 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.252540    4885 scope.go:122] "RemoveContainer" containerID="3a65fca9322b9e7260bb2d09e641a8e86e39bb6b48e7b5ee07772a9490156678"
Oct 30 03:45:12.259882 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.259857    4885 scope.go:122] "RemoveContainer" containerID="861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63"
Oct 30 03:45:12.260145 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:45:12.260126    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63\": container with ID starting with 861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63 not found: ID does not exist" containerID="861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63"
Oct 30 03:45:12.260204 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.260154    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63"} err="failed to get container status \"861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63\": rpc error: code = NotFound desc = could not find container \"861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63\": container with ID starting with 861882617b8011c00ca823890e1a444c4d1f72e235cead38aa2f0cfbaabfae63 not found: ID does not exist"
Oct 30 03:45:12.260204 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.260171    4885 scope.go:122] "RemoveContainer" containerID="3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697"
Oct 30 03:45:12.260438 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:45:12.260410    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697\": container with ID starting with 3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697 not found: ID does not exist" containerID="3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697"
Oct 30 03:45:12.260538 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.260439    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697"} err="failed to get container status \"3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697\": rpc error: code = NotFound desc = could not find container \"3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697\": container with ID starting with 3b03a0299077d2683da01f94c103072ae312119f23a865ce2b53e51b06d4f697 not found: ID does not exist"
Oct 30 03:45:12.260538 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.260451    4885 scope.go:122] "RemoveContainer" containerID="3a65fca9322b9e7260bb2d09e641a8e86e39bb6b48e7b5ee07772a9490156678"
Oct 30 03:45:12.260795 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:45:12.260774    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a65fca9322b9e7260bb2d09e641a8e86e39bb6b48e7b5ee07772a9490156678\": container with ID starting with 3a65fca9322b9e7260bb2d09e641a8e86e39bb6b48e7b5ee07772a9490156678 not found: ID does not exist" containerID="3a65fca9322b9e7260bb2d09e641a8e86e39bb6b48e7b5ee07772a9490156678"
Oct 30 03:45:12.260859 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.260801    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a65fca9322b9e7260bb2d09e641a8e86e39bb6b48e7b5ee07772a9490156678"} err="failed to get container status \"3a65fca9322b9e7260bb2d09e641a8e86e39bb6b48e7b5ee07772a9490156678\": rpc error: code = NotFound desc = could not find container \"3a65fca9322b9e7260bb2d09e641a8e86e39bb6b48e7b5ee07772a9490156678\": container with ID starting with 3a65fca9322b9e7260bb2d09e641a8e86e39bb6b48e7b5ee07772a9490156678 not found: ID does not exist"
Oct 30 03:45:12.280753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.280697    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"]
Oct 30 03:45:12.290543 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.290502    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-zl6kf"]
Oct 30 03:45:12.624073 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:12.624001    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" path="/var/lib/kubelet/pods/8487d62d-b1fb-435f-a00f-2b61bcfbf7cf/volumes"
Oct 30 03:45:14.465284 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.465221    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"]
Oct 30 03:45:14.465775 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.465694    4885 cpu_manager.go:414] "containerMap: removing container" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="storage-initializer"
Oct 30 03:45:14.465775 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.465709    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="storage-initializer"
Oct 30 03:45:14.465775 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.465742    4885 cpu_manager.go:414] "containerMap: removing container" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kube-rbac-proxy"
Oct 30 03:45:14.465775 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.465751    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kube-rbac-proxy"
Oct 30 03:45:14.465775 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.465762    4885 cpu_manager.go:414] "containerMap: removing container" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container"
Oct 30 03:45:14.465775 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.465771    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container"
Oct 30 03:45:14.465988 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.465853    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kube-rbac-proxy"
Oct 30 03:45:14.465988 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.465862    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="8487d62d-b1fb-435f-a00f-2b61bcfbf7cf" containerName="kserve-container"
Oct 30 03:45:14.468743 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.468726    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.481572 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.481554    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Oct 30 03:45:14.491590 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.491558    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:45:14.491590 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.491560    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:45:14.491590 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.491585    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:45:14.491793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.491585    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:45:14.491793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.491582    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:45:14.491793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.491687    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 30 03:45:14.493416 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.493400    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"]
Oct 30 03:45:14.646798 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.646766    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51af2bb2-1b91-4a23-b604-3a5893c3e13b-proxy-tls\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.646979 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.646824    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.646979 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.646845    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.646979 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.646904    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.646979 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.646969    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdps5\" (UniqueName: \"kubernetes.io/projected/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-api-access-pdps5\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.748197 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.748136    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51af2bb2-1b91-4a23-b604-3a5893c3e13b-proxy-tls\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.748197 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.748182    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.748401 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.748204    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.748401 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.748220    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.748401 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.748247    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdps5\" (UniqueName: \"kubernetes.io/projected/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-api-access-pdps5\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.748746 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.748721    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.748935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.748915    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.749002 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.748939    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.750790 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.750771    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51af2bb2-1b91-4a23-b604-3a5893c3e13b-proxy-tls\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.769645 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.769606    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdps5\" (UniqueName: \"kubernetes.io/projected/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-api-access-pdps5\") pod \"isvc-canary-rollback-predictor-5769c8765-vbmpl\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:14.782471 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:14.782449    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:15.087006 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:15.086891    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"]
Oct 30 03:45:15.089857 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:45:15.089824    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51af2bb2_1b91_4a23_b604_3a5893c3e13b.slice/crio-963a1ecb9dd9dfadcf5e158a7691b0b7df57086e91ffb6c538f1ff4f89e49c3e WatchSource:0}: Error finding container 963a1ecb9dd9dfadcf5e158a7691b0b7df57086e91ffb6c538f1ff4f89e49c3e: Status 404 returned error can't find the container with id 963a1ecb9dd9dfadcf5e158a7691b0b7df57086e91ffb6c538f1ff4f89e49c3e
Oct 30 03:45:15.251633 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:15.251581    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" event={"ID":"51af2bb2-1b91-4a23-b604-3a5893c3e13b","Type":"ContainerStarted","Data":"4aabd62eb6d3db5a3ba59906288e6b17e51e496dd23a0723ab8f2fb13b4da92b"}
Oct 30 03:45:15.251814 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:15.251662    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" event={"ID":"51af2bb2-1b91-4a23-b604-3a5893c3e13b","Type":"ContainerStarted","Data":"963a1ecb9dd9dfadcf5e158a7691b0b7df57086e91ffb6c538f1ff4f89e49c3e"}
Oct 30 03:45:15.965258 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:15.965234    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:45:19.271073 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:19.271042    4885 generic.go:356] "Generic (PLEG): container finished" podID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerID="4aabd62eb6d3db5a3ba59906288e6b17e51e496dd23a0723ab8f2fb13b4da92b" exitCode=0
Oct 30 03:45:19.271483 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:19.271105    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" event={"ID":"51af2bb2-1b91-4a23-b604-3a5893c3e13b","Type":"ContainerDied","Data":"4aabd62eb6d3db5a3ba59906288e6b17e51e496dd23a0723ab8f2fb13b4da92b"}
Oct 30 03:45:20.278650 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:20.278595    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" event={"ID":"51af2bb2-1b91-4a23-b604-3a5893c3e13b","Type":"ContainerStarted","Data":"8c57524d376937548da6b67ca59db68ab87a85e56cb9a9a27886c274e225249b"}
Oct 30 03:45:20.279090 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:20.278696    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" event={"ID":"51af2bb2-1b91-4a23-b604-3a5893c3e13b","Type":"ContainerStarted","Data":"51723e26bbb657f2e167a68a1409d67c9449cc390302acd77cbc668378d28275"}
Oct 30 03:45:20.279090 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:20.278978    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:20.279210 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:20.279133    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:20.280320 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:20.280295    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 03:45:20.330501 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:20.330455    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podStartSLOduration=2.35235736 podStartE2EDuration="6.330442941s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.978085581s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:20.329279462 +0000 UTC m=+3025.839448529" watchObservedRunningTime="2026-10-30 03:45:20.330442941 +0000 UTC m=+3025.840611967"
Oct 30 03:45:21.283443 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:21.283401    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 03:45:25.921198 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:45:25.921167    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:45:26.287301 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:26.287276    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:45:26.287783 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:26.287753    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 03:45:26.548380 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:26.548309    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"]
Oct 30 03:45:26.548571 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:26.548543    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kserve-container" containerID="cri-o://9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1" gracePeriod=30
Oct 30 03:45:26.548677 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:26.548584    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kube-rbac-proxy" containerID="cri-o://6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239" gracePeriod=30
Oct 30 03:45:27.311395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:27.311359    4885 generic.go:356] "Generic (PLEG): container finished" podID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerID="6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239" exitCode=0
Oct 30 03:45:27.311795 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:27.311429    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" event={"ID":"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd","Type":"ContainerDied","Data":"6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239"}
Oct 30 03:45:30.715710 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.715686    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:45:30.776190 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.776163    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-api-access-sq6m6\" (UniqueName: \"kubernetes.io/projected/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-api-access-sq6m6\") pod \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") "
Oct 30 03:45:30.776335 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.776233    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-proxy-tls\") pod \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") "
Oct 30 03:45:30.776335 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.776267    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-cabundle-cert\") pod \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") "
Oct 30 03:45:30.776335 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.776319    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-rbac-proxy-sar-config\") pod \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") "
Oct 30 03:45:30.776502 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.776352    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kserve-provision-location\") pod \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\" (UID: \"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd\") "
Oct 30 03:45:30.776774 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.776752    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-rbac-proxy-sar-config" pod "d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" (UID: "d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:30.776856 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.776796    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-cabundle-cert" pod "d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" (UID: "d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:30.776999 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.776969    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kserve-provision-location" pod "d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" (UID: "d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:30.778327 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.778305    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-api-access-sq6m6" pod "d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" (UID: "d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd"). InnerVolumeSpecName "kube-api-access-sq6m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:30.778415 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.778405    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-proxy-tls" pod "d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" (UID: "d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:30.876901 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.876875    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:45:30.876901 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.876896    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:45:30.877050 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.876906    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:45:30.877050 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.876916    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:45:30.877050 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:30.876924    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq6m6\" (UniqueName: \"kubernetes.io/projected/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd-kube-api-access-sq6m6\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:45:31.330486 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.330446    4885 generic.go:356] "Generic (PLEG): container finished" podID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerID="9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1" exitCode=0
Oct 30 03:45:31.330698 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.330485    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" event={"ID":"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd","Type":"ContainerDied","Data":"9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1"}
Oct 30 03:45:31.330698 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.330503    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"
Oct 30 03:45:31.330698 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.330549    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf" event={"ID":"d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd","Type":"ContainerDied","Data":"01f4cef2a211e7190462dfb2ffac038372866148aaa77e57ff016abf427a8278"}
Oct 30 03:45:31.330698 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.330576    4885 scope.go:122] "RemoveContainer" containerID="6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239"
Oct 30 03:45:31.340753 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.340733    4885 scope.go:122] "RemoveContainer" containerID="9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1"
Oct 30 03:45:31.348354 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.348338    4885 scope.go:122] "RemoveContainer" containerID="05ccc1dcad159222c08c9e948f010b795e8988514ba0c14c24657eea43049297"
Oct 30 03:45:31.355335 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.355316    4885 scope.go:122] "RemoveContainer" containerID="6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239"
Oct 30 03:45:31.355584 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:45:31.355568    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239\": container with ID starting with 6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239 not found: ID does not exist" containerID="6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239"
Oct 30 03:45:31.355656 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.355591    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239"} err="failed to get container status \"6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239\": rpc error: code = NotFound desc = could not find container \"6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239\": container with ID starting with 6cdc419bce812bfece231dd2c2a5903c067a63afd700eb573da3e3fbefbee239 not found: ID does not exist"
Oct 30 03:45:31.355656 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.355607    4885 scope.go:122] "RemoveContainer" containerID="9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1"
Oct 30 03:45:31.355826 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:45:31.355811    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1\": container with ID starting with 9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1 not found: ID does not exist" containerID="9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1"
Oct 30 03:45:31.355879 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.355829    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1"} err="failed to get container status \"9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1\": rpc error: code = NotFound desc = could not find container \"9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1\": container with ID starting with 9a4218b0b2fc2b60cb93dcc5e499429bb69d9cf6da05b45acb753a059ea744d1 not found: ID does not exist"
Oct 30 03:45:31.355879 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.355841    4885 scope.go:122] "RemoveContainer" containerID="05ccc1dcad159222c08c9e948f010b795e8988514ba0c14c24657eea43049297"
Oct 30 03:45:31.356067 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:45:31.356053    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ccc1dcad159222c08c9e948f010b795e8988514ba0c14c24657eea43049297\": container with ID starting with 05ccc1dcad159222c08c9e948f010b795e8988514ba0c14c24657eea43049297 not found: ID does not exist" containerID="05ccc1dcad159222c08c9e948f010b795e8988514ba0c14c24657eea43049297"
Oct 30 03:45:31.356109 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.356069    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ccc1dcad159222c08c9e948f010b795e8988514ba0c14c24657eea43049297"} err="failed to get container status \"05ccc1dcad159222c08c9e948f010b795e8988514ba0c14c24657eea43049297\": rpc error: code = NotFound desc = could not find container \"05ccc1dcad159222c08c9e948f010b795e8988514ba0c14c24657eea43049297\": container with ID starting with 05ccc1dcad159222c08c9e948f010b795e8988514ba0c14c24657eea43049297 not found: ID does not exist"
Oct 30 03:45:31.386695 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.386674    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"]
Oct 30 03:45:31.396799 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:31.396780    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-predictor-7b554dbc6c-84mqf"]
Oct 30 03:45:32.624714 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:32.624682    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" path="/var/lib/kubelet/pods/d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd/volumes"
Oct 30 03:45:34.172633 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.172575    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"]
Oct 30 03:45:34.173088 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.173035    4885 cpu_manager.go:414] "containerMap: removing container" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kube-rbac-proxy"
Oct 30 03:45:34.173088 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.173049    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kube-rbac-proxy"
Oct 30 03:45:34.173088 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.173059    4885 cpu_manager.go:414] "containerMap: removing container" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kserve-container"
Oct 30 03:45:34.173088 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.173065    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kserve-container"
Oct 30 03:45:34.173088 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.173079    4885 cpu_manager.go:414] "containerMap: removing container" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="storage-initializer"
Oct 30 03:45:34.173088 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.173086    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="storage-initializer"
Oct 30 03:45:34.173407 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.173160    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kube-rbac-proxy"
Oct 30 03:45:34.173407 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.173170    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24b5da3-18f5-4b1a-afb3-fd10a7bea3bd" containerName="kserve-container"
Oct 30 03:45:34.178242 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.178223    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.191320 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.191296    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 03:45:34.191419 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.191306    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:45:34.191419 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.191324    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:45:34.191419 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.191324    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Oct 30 03:45:34.201066 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.201040    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:45:34.201066 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.201051    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:45:34.201236 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.201206    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:45:34.203314 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.203294    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"]
Oct 30 03:45:34.203573 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.203554    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdmwd\" (UniqueName: \"kubernetes.io/projected/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-api-access-fdmwd\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.203652 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.203596    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.203652 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.203638    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.203773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.203754    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.203828 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.203813    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.304911 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.304870    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.305111 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.304922    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.305111 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.304962    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.305111 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.305002    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.305111 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.305055    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdmwd\" (UniqueName: \"kubernetes.io/projected/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-api-access-fdmwd\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.305416 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.305388    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.305643 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.305602    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.305758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.305666    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.307598 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.307580    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.326540 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.326511    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdmwd\" (UniqueName: \"kubernetes.io/projected/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-api-access-fdmwd\") pod \"isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.495921 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.495888    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:34.804064 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:34.804020    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"]
Oct 30 03:45:34.806569 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:45:34.806544    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9fa4df_e4a6_4feb_8158_a22cc5e14ad6.slice/crio-50fefb32127211a5c0a740b1601496fdf82a3804deb5f0b16018a1839b0fd5d9 WatchSource:0}: Error finding container 50fefb32127211a5c0a740b1601496fdf82a3804deb5f0b16018a1839b0fd5d9: Status 404 returned error can't find the container with id 50fefb32127211a5c0a740b1601496fdf82a3804deb5f0b16018a1839b0fd5d9
Oct 30 03:45:35.351262 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:35.351224    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" event={"ID":"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6","Type":"ContainerStarted","Data":"341ddcee23e28f279c05fe819a3bbfe977ce98919f7a49e9ca66e5159e9adce4"}
Oct 30 03:45:35.351653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:35.351284    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" event={"ID":"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6","Type":"ContainerStarted","Data":"50fefb32127211a5c0a740b1601496fdf82a3804deb5f0b16018a1839b0fd5d9"}
Oct 30 03:45:36.287767 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:36.287724    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 03:45:39.370948 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:39.370915    4885 generic.go:356] "Generic (PLEG): container finished" podID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerID="341ddcee23e28f279c05fe819a3bbfe977ce98919f7a49e9ca66e5159e9adce4" exitCode=0
Oct 30 03:45:39.371354 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:39.370988    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" event={"ID":"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6","Type":"ContainerDied","Data":"341ddcee23e28f279c05fe819a3bbfe977ce98919f7a49e9ca66e5159e9adce4"}
Oct 30 03:45:40.377774 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:40.377737    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" event={"ID":"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6","Type":"ContainerStarted","Data":"c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98"}
Oct 30 03:45:40.378222 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:40.377798    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" event={"ID":"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6","Type":"ContainerStarted","Data":"eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e"}
Oct 30 03:45:40.378222 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:40.378005    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:40.439946 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:40.439889    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" podStartSLOduration=2.487071868 podStartE2EDuration="6.439874426s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.952802558s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:40.428683783 +0000 UTC m=+3045.938852810" watchObservedRunningTime="2026-10-30 03:45:40.439874426 +0000 UTC m=+3045.950043442"
Oct 30 03:45:41.382868 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:41.382832    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:46.288399 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:46.288361    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 03:45:47.392747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:47.392714    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:45:55.921679 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:45:55.921642    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:45:56.287762 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:45:56.287719    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 03:46:06.288334 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:06.288288    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 03:46:16.287987 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:16.287942    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 03:46:17.397050 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:17.397018    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:46:24.388476 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:24.388441    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"]
Oct 30 03:46:24.389031 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:24.388729    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerName="kserve-container" containerID="cri-o://eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e" gracePeriod=30
Oct 30 03:46:24.389031 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:24.388743    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerName="kube-rbac-proxy" containerID="cri-o://c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98" gracePeriod=30
Oct 30 03:46:24.574296 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:24.574262    4885 generic.go:356] "Generic (PLEG): container finished" podID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerID="c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98" exitCode=0
Oct 30 03:46:24.574461 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:24.574330    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" event={"ID":"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6","Type":"ContainerDied","Data":"c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98"}
Oct 30 03:46:25.922153 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:46:25.922119    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:46:26.288635 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:26.288575    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 03:46:27.387370 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:27.387331    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.81:8643/healthz\": dial tcp 10.132.0.81:8643: connect: connection refused"
Oct 30 03:46:27.393822 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:27.393790    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.81:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.81:8080: connect: connection refused"
Oct 30 03:46:31.236082 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.236054    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:46:31.257115 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.257086    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-api-access-fdmwd\" (UniqueName: \"kubernetes.io/projected/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-api-access-fdmwd\") pod \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") "
Oct 30 03:46:31.257259 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.257124    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-rbac-proxy-sar-config\") pod \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") "
Oct 30 03:46:31.257259 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.257154    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kserve-provision-location\") pod \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") "
Oct 30 03:46:31.257492 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.257467    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-rbac-proxy-sar-config" pod "4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" (UID: "4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:31.257537 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.257521    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kserve-provision-location" pod "4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" (UID: "4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:31.259546 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.259521    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-api-access-fdmwd" pod "4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" (UID: "4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6"). InnerVolumeSpecName "kube-api-access-fdmwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:31.357714 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.357602    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-cabundle-cert\") pod \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") "
Oct 30 03:46:31.357714 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.357671    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-proxy-tls\") pod \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\" (UID: \"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6\") "
Oct 30 03:46:31.357968 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.357850    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdmwd\" (UniqueName: \"kubernetes.io/projected/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-api-access-fdmwd\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:46:31.357968 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.357866    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:46:31.357968 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.357876    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:46:31.357968 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.357927    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-cabundle-cert" pod "4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" (UID: "4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:31.360050 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.360026    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-proxy-tls" pod "4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" (UID: "4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:31.458771 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.458732    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:46:31.458771 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.458763    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:46:31.605996 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.605964    4885 generic.go:356] "Generic (PLEG): container finished" podID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerID="eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e" exitCode=0
Oct 30 03:46:31.606184 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.606021    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"
Oct 30 03:46:31.606184 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.606048    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" event={"ID":"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6","Type":"ContainerDied","Data":"eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e"}
Oct 30 03:46:31.606184 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.606107    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw" event={"ID":"4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6","Type":"ContainerDied","Data":"50fefb32127211a5c0a740b1601496fdf82a3804deb5f0b16018a1839b0fd5d9"}
Oct 30 03:46:31.606184 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.606133    4885 scope.go:122] "RemoveContainer" containerID="c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98"
Oct 30 03:46:31.614698 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.614677    4885 scope.go:122] "RemoveContainer" containerID="eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e"
Oct 30 03:46:31.623267 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.623247    4885 scope.go:122] "RemoveContainer" containerID="341ddcee23e28f279c05fe819a3bbfe977ce98919f7a49e9ca66e5159e9adce4"
Oct 30 03:46:31.630897 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.630868    4885 scope.go:122] "RemoveContainer" containerID="c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98"
Oct 30 03:46:31.631164 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:46:31.631144    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98\": container with ID starting with c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98 not found: ID does not exist" containerID="c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98"
Oct 30 03:46:31.631227 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.631174    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98"} err="failed to get container status \"c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98\": rpc error: code = NotFound desc = could not find container \"c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98\": container with ID starting with c715411543672a02b0fb04ca49614a3be4efcc1fcf70d2f37740c78a0380cb98 not found: ID does not exist"
Oct 30 03:46:31.631227 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.631193    4885 scope.go:122] "RemoveContainer" containerID="eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e"
Oct 30 03:46:31.631443 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:46:31.631426    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e\": container with ID starting with eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e not found: ID does not exist" containerID="eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e"
Oct 30 03:46:31.631489 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.631449    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e"} err="failed to get container status \"eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e\": rpc error: code = NotFound desc = could not find container \"eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e\": container with ID starting with eaab61cc13601ff61bf2bf70ab1b854cc3df9c7db0625a43b6aa6dc045b6556e not found: ID does not exist"
Oct 30 03:46:31.631489 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.631464    4885 scope.go:122] "RemoveContainer" containerID="341ddcee23e28f279c05fe819a3bbfe977ce98919f7a49e9ca66e5159e9adce4"
Oct 30 03:46:31.631709 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:46:31.631690    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341ddcee23e28f279c05fe819a3bbfe977ce98919f7a49e9ca66e5159e9adce4\": container with ID starting with 341ddcee23e28f279c05fe819a3bbfe977ce98919f7a49e9ca66e5159e9adce4 not found: ID does not exist" containerID="341ddcee23e28f279c05fe819a3bbfe977ce98919f7a49e9ca66e5159e9adce4"
Oct 30 03:46:31.631766 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.631713    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341ddcee23e28f279c05fe819a3bbfe977ce98919f7a49e9ca66e5159e9adce4"} err="failed to get container status \"341ddcee23e28f279c05fe819a3bbfe977ce98919f7a49e9ca66e5159e9adce4\": rpc error: code = NotFound desc = could not find container \"341ddcee23e28f279c05fe819a3bbfe977ce98919f7a49e9ca66e5159e9adce4\": container with ID starting with 341ddcee23e28f279c05fe819a3bbfe977ce98919f7a49e9ca66e5159e9adce4 not found: ID does not exist"
Oct 30 03:46:31.652117 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.652089    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"]
Oct 30 03:46:31.662157 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:31.662131    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-mlserver-predictor-77cd89fff7-7x6mw"]
Oct 30 03:46:32.624167 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:32.624135    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" path="/var/lib/kubelet/pods/4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6/volumes"
Oct 30 03:46:32.625071 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:32.625046    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 30 03:46:34.059248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.059212    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"]
Oct 30 03:46:34.059666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.059552    4885 cpu_manager.go:414] "containerMap: removing container" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerName="kserve-container"
Oct 30 03:46:34.059666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.059563    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerName="kserve-container"
Oct 30 03:46:34.059666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.059584    4885 cpu_manager.go:414] "containerMap: removing container" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerName="storage-initializer"
Oct 30 03:46:34.059666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.059591    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerName="storage-initializer"
Oct 30 03:46:34.059666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.059624    4885 cpu_manager.go:414] "containerMap: removing container" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerName="kube-rbac-proxy"
Oct 30 03:46:34.059666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.059630    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerName="kube-rbac-proxy"
Oct 30 03:46:34.059997 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.059683    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerName="kube-rbac-proxy"
Oct 30 03:46:34.059997 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.059690    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d9fa4df-e4a6-4feb-8158-a22cc5e14ad6" containerName="kserve-container"
Oct 30 03:46:34.065509 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.065487    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.078060 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.078033    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:46:34.081259 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.081238    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.081356 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.081268    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.081411 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.081389    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd4n7\" (UniqueName: \"kubernetes.io/projected/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-api-access-pd4n7\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.081450 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.081422    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69f1f07c-5671-44e3-ab6f-48cceefddd90-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.081450 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.081444    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f1f07c-5671-44e3-ab6f-48cceefddd90-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.088135 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.088111    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:46:34.088281 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.088144    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:46:34.088281 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.088157    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:46:34.088448 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.088158    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Oct 30 03:46:34.090358 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.090337    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"]
Oct 30 03:46:34.178869 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.178844    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:46:34.178869 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.178842    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Oct 30 03:46:34.182152 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.182120    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.182221 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.182160    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.182221 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.182203    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4n7\" (UniqueName: \"kubernetes.io/projected/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-api-access-pd4n7\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.182284 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.182220    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69f1f07c-5671-44e3-ab6f-48cceefddd90-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.182284 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.182239    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f1f07c-5671-44e3-ab6f-48cceefddd90-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.182715 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.182686    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f1f07c-5671-44e3-ab6f-48cceefddd90-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.182882 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.182859    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.182985 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.182963    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.184814 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.184795    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69f1f07c-5671-44e3-ab6f-48cceefddd90-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.203252 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.203232    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd4n7\" (UniqueName: \"kubernetes.io/projected/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-api-access-pd4n7\") pod \"xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") " pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.383669 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.383555    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:34.691024 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:34.690876    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"]
Oct 30 03:46:34.693173 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:46:34.693141    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f1f07c_5671_44e3_ab6f_48cceefddd90.slice/crio-96661216bae4780c75bd849b89dfc054c1627bf2723c17fa6ba618db3508bec6 WatchSource:0}: Error finding container 96661216bae4780c75bd849b89dfc054c1627bf2723c17fa6ba618db3508bec6: Status 404 returned error can't find the container with id 96661216bae4780c75bd849b89dfc054c1627bf2723c17fa6ba618db3508bec6
Oct 30 03:46:35.625930 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:35.625895    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" event={"ID":"69f1f07c-5671-44e3-ab6f-48cceefddd90","Type":"ContainerStarted","Data":"cecbcfe74b5cb20436285f9ff40a1c945f96545927c64112566bfb3aa920a27f"}
Oct 30 03:46:35.626331 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:35.625952    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" event={"ID":"69f1f07c-5671-44e3-ab6f-48cceefddd90","Type":"ContainerStarted","Data":"96661216bae4780c75bd849b89dfc054c1627bf2723c17fa6ba618db3508bec6"}
Oct 30 03:46:38.639315 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:38.639287    4885 generic.go:356] "Generic (PLEG): container finished" podID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerID="cecbcfe74b5cb20436285f9ff40a1c945f96545927c64112566bfb3aa920a27f" exitCode=0
Oct 30 03:46:38.639669 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:38.639359    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" event={"ID":"69f1f07c-5671-44e3-ab6f-48cceefddd90","Type":"ContainerDied","Data":"cecbcfe74b5cb20436285f9ff40a1c945f96545927c64112566bfb3aa920a27f"}
Oct 30 03:46:39.646197 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:39.646163    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" event={"ID":"69f1f07c-5671-44e3-ab6f-48cceefddd90","Type":"ContainerStarted","Data":"de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032"}
Oct 30 03:46:39.646713 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:39.646235    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" event={"ID":"69f1f07c-5671-44e3-ab6f-48cceefddd90","Type":"ContainerStarted","Data":"d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657"}
Oct 30 03:46:39.646713 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:39.646457    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:39.708911 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:39.708867    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" podStartSLOduration=1.96963064 podStartE2EDuration="5.708855036s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.739224396s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:39.698533352 +0000 UTC m=+3105.208702389" watchObservedRunningTime="2026-10-30 03:46:39.708855036 +0000 UTC m=+3105.219024061"
Oct 30 03:46:40.651287 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:40.651251    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:42.626264 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:42.626228    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:46:44.767896 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:44.767858    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"]
Oct 30 03:46:44.768341 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:44.768092    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container" containerID="cri-o://51723e26bbb657f2e167a68a1409d67c9449cc390302acd77cbc668378d28275" gracePeriod=30
Oct 30 03:46:44.768423 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:44.768402    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kube-rbac-proxy" containerID="cri-o://8c57524d376937548da6b67ca59db68ab87a85e56cb9a9a27886c274e225249b" gracePeriod=30
Oct 30 03:46:45.673596 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:45.673564    4885 generic.go:356] "Generic (PLEG): container finished" podID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerID="8c57524d376937548da6b67ca59db68ab87a85e56cb9a9a27886c274e225249b" exitCode=0
Oct 30 03:46:45.673785 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:45.673637    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" event={"ID":"51af2bb2-1b91-4a23-b604-3a5893c3e13b","Type":"ContainerDied","Data":"8c57524d376937548da6b67ca59db68ab87a85e56cb9a9a27886c274e225249b"}
Oct 30 03:46:46.284129 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:46.284093    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.80:8643/healthz\": dial tcp 10.132.0.80:8643: connect: connection refused"
Oct 30 03:46:46.661518 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:46.661448    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:46:49.693838 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.693811    4885 generic.go:356] "Generic (PLEG): container finished" podID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerID="51723e26bbb657f2e167a68a1409d67c9449cc390302acd77cbc668378d28275" exitCode=0
Oct 30 03:46:49.694149 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.693893    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" event={"ID":"51af2bb2-1b91-4a23-b604-3a5893c3e13b","Type":"ContainerDied","Data":"51723e26bbb657f2e167a68a1409d67c9449cc390302acd77cbc668378d28275"}
Oct 30 03:46:49.818960 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.818937    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:46:49.909932 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.909866    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51af2bb2-1b91-4a23-b604-3a5893c3e13b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/51af2bb2-1b91-4a23-b604-3a5893c3e13b-proxy-tls\") pod \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") "
Oct 30 03:46:49.909932 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.909910    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-cabundle-cert\") pod \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") "
Oct 30 03:46:49.910149 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.909942    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-rbac-proxy-sar-config\") pod \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") "
Oct 30 03:46:49.910149 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.909976    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kserve-provision-location\") pod \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") "
Oct 30 03:46:49.910149 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.910035    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-api-access-pdps5\" (UniqueName: \"kubernetes.io/projected/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-api-access-pdps5\") pod \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\" (UID: \"51af2bb2-1b91-4a23-b604-3a5893c3e13b\") "
Oct 30 03:46:49.910316 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.910291    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-cabundle-cert" pod "51af2bb2-1b91-4a23-b604-3a5893c3e13b" (UID: "51af2bb2-1b91-4a23-b604-3a5893c3e13b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:49.910316 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.910304    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kserve-provision-location" pod "51af2bb2-1b91-4a23-b604-3a5893c3e13b" (UID: "51af2bb2-1b91-4a23-b604-3a5893c3e13b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:49.910383 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.910345    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-rbac-proxy-sar-config" pod "51af2bb2-1b91-4a23-b604-3a5893c3e13b" (UID: "51af2bb2-1b91-4a23-b604-3a5893c3e13b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:49.912171 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.912153    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51af2bb2-1b91-4a23-b604-3a5893c3e13b-proxy-tls" pod "51af2bb2-1b91-4a23-b604-3a5893c3e13b" (UID: "51af2bb2-1b91-4a23-b604-3a5893c3e13b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:49.912272 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:49.912255    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-api-access-pdps5" pod "51af2bb2-1b91-4a23-b604-3a5893c3e13b" (UID: "51af2bb2-1b91-4a23-b604-3a5893c3e13b"). InnerVolumeSpecName "kube-api-access-pdps5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:50.011591 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:50.011563    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdps5\" (UniqueName: \"kubernetes.io/projected/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-api-access-pdps5\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:46:50.011591 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:50.011588    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51af2bb2-1b91-4a23-b604-3a5893c3e13b-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:46:50.011591 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:50.011599    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:46:50.011802 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:50.011607    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:46:50.011802 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:50.011631    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51af2bb2-1b91-4a23-b604-3a5893c3e13b-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:46:50.700151 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:50.700118    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl" event={"ID":"51af2bb2-1b91-4a23-b604-3a5893c3e13b","Type":"ContainerDied","Data":"963a1ecb9dd9dfadcf5e158a7691b0b7df57086e91ffb6c538f1ff4f89e49c3e"}
Oct 30 03:46:50.700151 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:50.700149    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"
Oct 30 03:46:50.700718 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:50.700186    4885 scope.go:122] "RemoveContainer" containerID="8c57524d376937548da6b67ca59db68ab87a85e56cb9a9a27886c274e225249b"
Oct 30 03:46:50.708653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:50.708637    4885 scope.go:122] "RemoveContainer" containerID="51723e26bbb657f2e167a68a1409d67c9449cc390302acd77cbc668378d28275"
Oct 30 03:46:50.719738 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:50.719718    4885 scope.go:122] "RemoveContainer" containerID="4aabd62eb6d3db5a3ba59906288e6b17e51e496dd23a0723ab8f2fb13b4da92b"
Oct 30 03:46:50.756393 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:50.756371    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"]
Oct 30 03:46:50.766462 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:50.766439    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-vbmpl"]
Oct 30 03:46:52.624452 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.624422    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" path="/var/lib/kubelet/pods/51af2bb2-1b91-4a23-b604-3a5893c3e13b/volumes"
Oct 30 03:46:52.630860 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.630835    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"]
Oct 30 03:46:52.631199 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.631183    4885 cpu_manager.go:414] "containerMap: removing container" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="storage-initializer"
Oct 30 03:46:52.631285 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.631202    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="storage-initializer"
Oct 30 03:46:52.631285 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.631239    4885 cpu_manager.go:414] "containerMap: removing container" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kube-rbac-proxy"
Oct 30 03:46:52.631285 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.631247    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kube-rbac-proxy"
Oct 30 03:46:52.631285 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.631278    4885 cpu_manager.go:414] "containerMap: removing container" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container"
Oct 30 03:46:52.631533 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.631286    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container"
Oct 30 03:46:52.631533 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.631368    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kube-rbac-proxy"
Oct 30 03:46:52.631533 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.631424    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="51af2bb2-1b91-4a23-b604-3a5893c3e13b" containerName="kserve-container"
Oct 30 03:46:52.636291 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.636270    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.649147 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.649123    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:46:52.659127 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.659056    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:46:52.659127 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.659083    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:46:52.659127 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.659097    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-stop-predictor-serving-cert\""
Oct 30 03:46:52.659127 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.659107    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Oct 30 03:46:52.659127 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.659112    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:46:52.659381 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.659205    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:46:52.660771 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.660750    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"]
Oct 30 03:46:52.733125 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.733093    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/001e418c-7e23-43d4-8264-d91e5eff2e94-proxy-tls\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.733125 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.733126    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-cabundle-cert\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.733374 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.733149    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g4qq\" (UniqueName: \"kubernetes.io/projected/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-api-access-6g4qq\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.733374 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.733292    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/001e418c-7e23-43d4-8264-d91e5eff2e94-kserve-provision-location\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.733483 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.733393    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.834786 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.834757    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/001e418c-7e23-43d4-8264-d91e5eff2e94-proxy-tls\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.834786 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.834789    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-cabundle-cert\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.835039 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.834814    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g4qq\" (UniqueName: \"kubernetes.io/projected/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-api-access-6g4qq\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.835039 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.834840    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/001e418c-7e23-43d4-8264-d91e5eff2e94-kserve-provision-location\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.835039 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.834889    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.835354 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.835327    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/001e418c-7e23-43d4-8264-d91e5eff2e94-kserve-provision-location\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.835566 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.835542    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.835800 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.835779    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-cabundle-cert\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.837426 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.837403    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/001e418c-7e23-43d4-8264-d91e5eff2e94-proxy-tls\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.855735 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.855713    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g4qq\" (UniqueName: \"kubernetes.io/projected/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-api-access-6g4qq\") pod \"isvc-canary-stop-predictor-574b8f9c57-thcj7\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:52.954966 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:52.954935    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:53.256909 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:53.256831    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"]
Oct 30 03:46:53.259687 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:46:53.259655    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001e418c_7e23_43d4_8264_d91e5eff2e94.slice/crio-0f082912c7afa8e537f5a2402b1c60019cbacb49213a099955cf5f3a2b967f29 WatchSource:0}: Error finding container 0f082912c7afa8e537f5a2402b1c60019cbacb49213a099955cf5f3a2b967f29: Status 404 returned error can't find the container with id 0f082912c7afa8e537f5a2402b1c60019cbacb49213a099955cf5f3a2b967f29
Oct 30 03:46:53.717820 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:53.717784    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" event={"ID":"001e418c-7e23-43d4-8264-d91e5eff2e94","Type":"ContainerStarted","Data":"bb05491321f3135fc02eeb30389ddb37624a8793f4a558d0570b5ea71a6516ae"}
Oct 30 03:46:53.718309 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:53.717859    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" event={"ID":"001e418c-7e23-43d4-8264-d91e5eff2e94","Type":"ContainerStarted","Data":"0f082912c7afa8e537f5a2402b1c60019cbacb49213a099955cf5f3a2b967f29"}
Oct 30 03:46:55.921771 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:46:55.921734    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:46:57.738009 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:57.737978    4885 generic.go:356] "Generic (PLEG): container finished" podID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerID="bb05491321f3135fc02eeb30389ddb37624a8793f4a558d0570b5ea71a6516ae" exitCode=0
Oct 30 03:46:57.738485 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:57.738020    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" event={"ID":"001e418c-7e23-43d4-8264-d91e5eff2e94","Type":"ContainerDied","Data":"bb05491321f3135fc02eeb30389ddb37624a8793f4a558d0570b5ea71a6516ae"}
Oct 30 03:46:58.744154 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:58.744122    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" event={"ID":"001e418c-7e23-43d4-8264-d91e5eff2e94","Type":"ContainerStarted","Data":"b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134"}
Oct 30 03:46:58.744536 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:58.744180    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" event={"ID":"001e418c-7e23-43d4-8264-d91e5eff2e94","Type":"ContainerStarted","Data":"e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e"}
Oct 30 03:46:58.744536 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:58.744484    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:58.744654 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:58.744637    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:46:58.746232 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:58.746206    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 03:46:58.795375 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:58.795332    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podStartSLOduration=2.943618975 podStartE2EDuration="6.795318177s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.851699202s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:58.794365272 +0000 UTC m=+3124.304534311" watchObservedRunningTime="2026-10-30 03:46:58.795318177 +0000 UTC m=+3124.305487202"
Oct 30 03:46:59.748717 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:46:59.748677    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 03:47:04.753411 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:04.753383    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:47:04.753953 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:04.753928    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 03:47:14.754439 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:14.754403    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 03:47:16.665515 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:16.665488    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:47:24.304996 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:24.304961    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"]
Oct 30 03:47:24.305407 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:24.305251    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerName="kserve-container" containerID="cri-o://d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657" gracePeriod=30
Oct 30 03:47:24.305407 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:24.305309    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerName="kube-rbac-proxy" containerID="cri-o://de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032" gracePeriod=30
Oct 30 03:47:24.753988 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:24.753952    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 03:47:24.853751 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:24.853718    4885 generic.go:356] "Generic (PLEG): container finished" podID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerID="de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032" exitCode=0
Oct 30 03:47:24.853928 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:24.853785    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" event={"ID":"69f1f07c-5671-44e3-ab6f-48cceefddd90","Type":"ContainerDied","Data":"de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032"}
Oct 30 03:47:25.921584 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:47:25.921531    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:47:26.656342 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:26.656308    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.82:8643/healthz\": dial tcp 10.132.0.82:8643: connect: connection refused"
Oct 30 03:47:26.662325 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:26.662297    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.82:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.82:8080: connect: connection refused"
Oct 30 03:47:30.452186 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.452163    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:47:30.537973 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.537893    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-cabundle-cert\") pod \"69f1f07c-5671-44e3-ab6f-48cceefddd90\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") "
Oct 30 03:47:30.537973 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.537946    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-api-access-pd4n7\" (UniqueName: \"kubernetes.io/projected/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-api-access-pd4n7\") pod \"69f1f07c-5671-44e3-ab6f-48cceefddd90\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") "
Oct 30 03:47:30.538202 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.537987    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f1f07c-5671-44e3-ab6f-48cceefddd90-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f1f07c-5671-44e3-ab6f-48cceefddd90-kserve-provision-location\") pod \"69f1f07c-5671-44e3-ab6f-48cceefddd90\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") "
Oct 30 03:47:30.538202 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.538025    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-rbac-proxy-sar-config\") pod \"69f1f07c-5671-44e3-ab6f-48cceefddd90\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") "
Oct 30 03:47:30.538202 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.538064    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69f1f07c-5671-44e3-ab6f-48cceefddd90-proxy-tls\" (UniqueName: \"kubernetes.io/secret/69f1f07c-5671-44e3-ab6f-48cceefddd90-proxy-tls\") pod \"69f1f07c-5671-44e3-ab6f-48cceefddd90\" (UID: \"69f1f07c-5671-44e3-ab6f-48cceefddd90\") "
Oct 30 03:47:30.538372 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.538301    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f1f07c-5671-44e3-ab6f-48cceefddd90-kserve-provision-location" pod "69f1f07c-5671-44e3-ab6f-48cceefddd90" (UID: "69f1f07c-5671-44e3-ab6f-48cceefddd90"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:47:30.538372 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.538271    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-cabundle-cert" pod "69f1f07c-5671-44e3-ab6f-48cceefddd90" (UID: "69f1f07c-5671-44e3-ab6f-48cceefddd90"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:30.538488 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.538424    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-rbac-proxy-sar-config" pod "69f1f07c-5671-44e3-ab6f-48cceefddd90" (UID: "69f1f07c-5671-44e3-ab6f-48cceefddd90"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:30.540242 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.540216    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f1f07c-5671-44e3-ab6f-48cceefddd90-proxy-tls" pod "69f1f07c-5671-44e3-ab6f-48cceefddd90" (UID: "69f1f07c-5671-44e3-ab6f-48cceefddd90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:30.540378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.540319    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-api-access-pd4n7" pod "69f1f07c-5671-44e3-ab6f-48cceefddd90" (UID: "69f1f07c-5671-44e3-ab6f-48cceefddd90"). InnerVolumeSpecName "kube-api-access-pd4n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:30.639322 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.639283    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69f1f07c-5671-44e3-ab6f-48cceefddd90-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:47:30.639322 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.639308    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:47:30.639322 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.639318    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69f1f07c-5671-44e3-ab6f-48cceefddd90-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:47:30.639322 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.639328    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69f1f07c-5671-44e3-ab6f-48cceefddd90-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:47:30.639657 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.639337    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd4n7\" (UniqueName: \"kubernetes.io/projected/69f1f07c-5671-44e3-ab6f-48cceefddd90-kube-api-access-pd4n7\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:47:30.881214 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.881134    4885 generic.go:356] "Generic (PLEG): container finished" podID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerID="d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657" exitCode=0
Oct 30 03:47:30.881214 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.881200    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"
Oct 30 03:47:30.881433 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.881216    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" event={"ID":"69f1f07c-5671-44e3-ab6f-48cceefddd90","Type":"ContainerDied","Data":"d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657"}
Oct 30 03:47:30.881433 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.881273    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl" event={"ID":"69f1f07c-5671-44e3-ab6f-48cceefddd90","Type":"ContainerDied","Data":"96661216bae4780c75bd849b89dfc054c1627bf2723c17fa6ba618db3508bec6"}
Oct 30 03:47:30.881433 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.881300    4885 scope.go:122] "RemoveContainer" containerID="de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032"
Oct 30 03:47:30.889583 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.889564    4885 scope.go:122] "RemoveContainer" containerID="d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657"
Oct 30 03:47:30.897082 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.897064    4885 scope.go:122] "RemoveContainer" containerID="cecbcfe74b5cb20436285f9ff40a1c945f96545927c64112566bfb3aa920a27f"
Oct 30 03:47:30.904250 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.904223    4885 scope.go:122] "RemoveContainer" containerID="de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032"
Oct 30 03:47:30.904510 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:47:30.904483    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032\": container with ID starting with de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032 not found: ID does not exist" containerID="de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032"
Oct 30 03:47:30.904556 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.904518    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032"} err="failed to get container status \"de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032\": rpc error: code = NotFound desc = could not find container \"de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032\": container with ID starting with de2839ebd7f207530fcd9233a3bf4082c2319d3067100448345802dcee234032 not found: ID does not exist"
Oct 30 03:47:30.904556 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.904537    4885 scope.go:122] "RemoveContainer" containerID="d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657"
Oct 30 03:47:30.904791 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:47:30.904766    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657\": container with ID starting with d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657 not found: ID does not exist" containerID="d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657"
Oct 30 03:47:30.904842 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.904797    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657"} err="failed to get container status \"d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657\": rpc error: code = NotFound desc = could not find container \"d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657\": container with ID starting with d8bfead6baf1231ed3b7c2da0183fc3a07bc3ee1c9c952faa618b6ba462e0657 not found: ID does not exist"
Oct 30 03:47:30.904842 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.904811    4885 scope.go:122] "RemoveContainer" containerID="cecbcfe74b5cb20436285f9ff40a1c945f96545927c64112566bfb3aa920a27f"
Oct 30 03:47:30.905019 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:47:30.905005    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cecbcfe74b5cb20436285f9ff40a1c945f96545927c64112566bfb3aa920a27f\": container with ID starting with cecbcfe74b5cb20436285f9ff40a1c945f96545927c64112566bfb3aa920a27f not found: ID does not exist" containerID="cecbcfe74b5cb20436285f9ff40a1c945f96545927c64112566bfb3aa920a27f"
Oct 30 03:47:30.905058 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.905023    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecbcfe74b5cb20436285f9ff40a1c945f96545927c64112566bfb3aa920a27f"} err="failed to get container status \"cecbcfe74b5cb20436285f9ff40a1c945f96545927c64112566bfb3aa920a27f\": rpc error: code = NotFound desc = could not find container \"cecbcfe74b5cb20436285f9ff40a1c945f96545927c64112566bfb3aa920a27f\": container with ID starting with cecbcfe74b5cb20436285f9ff40a1c945f96545927c64112566bfb3aa920a27f not found: ID does not exist"
Oct 30 03:47:30.926703 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.926678    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"]
Oct 30 03:47:30.930477 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:30.930460    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/xgboost-v2-mlserver-predictor-5dcfb76568-j8ftl"]
Oct 30 03:47:31.900908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.900864    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"]
Oct 30 03:47:31.901362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.901204    4885 cpu_manager.go:414] "containerMap: removing container" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerName="storage-initializer"
Oct 30 03:47:31.901362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.901215    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerName="storage-initializer"
Oct 30 03:47:31.901362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.901229    4885 cpu_manager.go:414] "containerMap: removing container" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerName="kube-rbac-proxy"
Oct 30 03:47:31.901362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.901234    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerName="kube-rbac-proxy"
Oct 30 03:47:31.901362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.901248    4885 cpu_manager.go:414] "containerMap: removing container" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerName="kserve-container"
Oct 30 03:47:31.901362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.901253    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerName="kserve-container"
Oct 30 03:47:31.901362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.901306    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerName="kserve-container"
Oct 30 03:47:31.901362 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.901318    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" containerName="kube-rbac-proxy"
Oct 30 03:47:31.906228 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.906209    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:31.919548 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.919530    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:47:31.929139 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.929085    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:47:31.929263 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.929236    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:47:31.929331 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.929273    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:47:31.929331 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.929280    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Oct 30 03:47:31.929420 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.929327    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:47:31.929420 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.929400    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Oct 30 03:47:31.930715 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:31.930695    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"]
Oct 30 03:47:32.050346 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.050316    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww7sl\" (UniqueName: \"kubernetes.io/projected/c104160b-0599-4481-ab57-86c3a25edaca-kube-api-access-ww7sl\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.050346 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.050349    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.050556 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.050369    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c104160b-0599-4481-ab57-86c3a25edaca-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.050556 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.050393    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.050556 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.050546    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c104160b-0599-4481-ab57-86c3a25edaca-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.151645 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.151562    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.151645 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.151632    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c104160b-0599-4481-ab57-86c3a25edaca-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.151897 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.151710    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7sl\" (UniqueName: \"kubernetes.io/projected/c104160b-0599-4481-ab57-86c3a25edaca-kube-api-access-ww7sl\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.151897 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.151739    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.151897 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.151765    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c104160b-0599-4481-ab57-86c3a25edaca-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.152118 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.152093    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c104160b-0599-4481-ab57-86c3a25edaca-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.152374 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.152347    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.152470 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.152448    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.154366 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.154349    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c104160b-0599-4481-ab57-86c3a25edaca-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.173216 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.173194    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww7sl\" (UniqueName: \"kubernetes.io/projected/c104160b-0599-4481-ab57-86c3a25edaca-kube-api-access-ww7sl\") pod \"isvc-xgboost-runtime-predictor-68fd78d655-td9bd\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.223086 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.223059    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:32.527559 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.527531    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"]
Oct 30 03:47:32.530348 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:47:32.530317    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc104160b_0599_4481_ab57_86c3a25edaca.slice/crio-e928d2b93f295c38194b9ad3d90fb7073443a3db08d53eca8cdc32d3fce11325 WatchSource:0}: Error finding container e928d2b93f295c38194b9ad3d90fb7073443a3db08d53eca8cdc32d3fce11325: Status 404 returned error can't find the container with id e928d2b93f295c38194b9ad3d90fb7073443a3db08d53eca8cdc32d3fce11325
Oct 30 03:47:32.625701 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.625671    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f1f07c-5671-44e3-ab6f-48cceefddd90" path="/var/lib/kubelet/pods/69f1f07c-5671-44e3-ab6f-48cceefddd90/volumes"
Oct 30 03:47:32.898306 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.898227    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" event={"ID":"c104160b-0599-4481-ab57-86c3a25edaca","Type":"ContainerStarted","Data":"df63f78ac0af92be4b5b2e50cfc92f3d0759a08ae3ccfabec80261b3437279fa"}
Oct 30 03:47:32.898306 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:32.898290    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" event={"ID":"c104160b-0599-4481-ab57-86c3a25edaca","Type":"ContainerStarted","Data":"e928d2b93f295c38194b9ad3d90fb7073443a3db08d53eca8cdc32d3fce11325"}
Oct 30 03:47:34.754593 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:34.754559    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 03:47:36.917426 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:36.917341    4885 generic.go:356] "Generic (PLEG): container finished" podID="c104160b-0599-4481-ab57-86c3a25edaca" containerID="df63f78ac0af92be4b5b2e50cfc92f3d0759a08ae3ccfabec80261b3437279fa" exitCode=0
Oct 30 03:47:36.917426 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:36.917401    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" event={"ID":"c104160b-0599-4481-ab57-86c3a25edaca","Type":"ContainerDied","Data":"df63f78ac0af92be4b5b2e50cfc92f3d0759a08ae3ccfabec80261b3437279fa"}
Oct 30 03:47:37.924735 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:37.924703    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" event={"ID":"c104160b-0599-4481-ab57-86c3a25edaca","Type":"ContainerStarted","Data":"3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46"}
Oct 30 03:47:37.925163 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:37.924770    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" event={"ID":"c104160b-0599-4481-ab57-86c3a25edaca","Type":"ContainerStarted","Data":"c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3"}
Oct 30 03:47:37.925163 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:37.925047    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:37.925276 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:37.925187    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:37.926544 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:37.926518    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.84:8080: connect: connection refused"
Oct 30 03:47:37.976068 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:37.976024    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podStartSLOduration=3.006875029 podStartE2EDuration="6.976011932s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.969136903s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:37.975183326 +0000 UTC m=+3163.485352369" watchObservedRunningTime="2026-10-30 03:47:37.976011932 +0000 UTC m=+3163.486180973"
Oct 30 03:47:38.929497 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:38.929455    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.84:8080: connect: connection refused"
Oct 30 03:47:43.934389 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:43.934355    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:47:43.934984 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:43.934957    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.84:8080: connect: connection refused"
Oct 30 03:47:44.754225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:44.754191    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 03:47:53.935903 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:53.935858    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.84:8080: connect: connection refused"
Oct 30 03:47:54.754550 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:47:54.754516    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 03:47:55.922749 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:47:55.922713    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:48:03.934899 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:03.934854    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.84:8080: connect: connection refused"
Oct 30 03:48:04.753860 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:04.753818    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 03:48:13.935773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:13.935679    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.84:8080: connect: connection refused"
Oct 30 03:48:14.754629 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:14.754593    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:48:23.935266 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:23.935222    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.84:8080: connect: connection refused"
Oct 30 03:48:25.922338 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:48:25.922301    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:48:32.651207 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:32.651173    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"]
Oct 30 03:48:32.651807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:32.651402    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container" containerID="cri-o://e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e" gracePeriod=30
Oct 30 03:48:32.651807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:32.651439    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kube-rbac-proxy" containerID="cri-o://b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134" gracePeriod=30
Oct 30 03:48:33.172746 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:33.172712    4885 generic.go:356] "Generic (PLEG): container finished" podID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerID="b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134" exitCode=0
Oct 30 03:48:33.172949 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:33.172794    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" event={"ID":"001e418c-7e23-43d4-8264-d91e5eff2e94","Type":"ContainerDied","Data":"b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134"}
Oct 30 03:48:33.935466 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:33.935420    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.84:8080: connect: connection refused"
Oct 30 03:48:34.749003 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:34.748964    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.83:8643/healthz\": dial tcp 10.132.0.83:8643: connect: connection refused"
Oct 30 03:48:34.754312 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:34.754288    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 30 03:48:37.896160 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:37.896135    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:48:37.986573 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:37.986546    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-cabundle-cert\") pod \"001e418c-7e23-43d4-8264-d91e5eff2e94\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") "
Oct 30 03:48:37.986769 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:37.986583    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/001e418c-7e23-43d4-8264-d91e5eff2e94-proxy-tls\" (UniqueName: \"kubernetes.io/secret/001e418c-7e23-43d4-8264-d91e5eff2e94-proxy-tls\") pod \"001e418c-7e23-43d4-8264-d91e5eff2e94\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") "
Oct 30 03:48:37.986769 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:37.986655    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-rbac-proxy-sar-config\") pod \"001e418c-7e23-43d4-8264-d91e5eff2e94\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") "
Oct 30 03:48:37.986769 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:37.986703    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-api-access-6g4qq\" (UniqueName: \"kubernetes.io/projected/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-api-access-6g4qq\") pod \"001e418c-7e23-43d4-8264-d91e5eff2e94\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") "
Oct 30 03:48:37.986769 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:37.986724    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/001e418c-7e23-43d4-8264-d91e5eff2e94-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/001e418c-7e23-43d4-8264-d91e5eff2e94-kserve-provision-location\") pod \"001e418c-7e23-43d4-8264-d91e5eff2e94\" (UID: \"001e418c-7e23-43d4-8264-d91e5eff2e94\") "
Oct 30 03:48:37.987128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:37.987070    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-cabundle-cert" pod "001e418c-7e23-43d4-8264-d91e5eff2e94" (UID: "001e418c-7e23-43d4-8264-d91e5eff2e94"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:37.987128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:37.987112    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001e418c-7e23-43d4-8264-d91e5eff2e94-kserve-provision-location" pod "001e418c-7e23-43d4-8264-d91e5eff2e94" (UID: "001e418c-7e23-43d4-8264-d91e5eff2e94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:48:37.987255 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:37.987165    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-rbac-proxy-sar-config" pod "001e418c-7e23-43d4-8264-d91e5eff2e94" (UID: "001e418c-7e23-43d4-8264-d91e5eff2e94"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:37.988826 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:37.988807    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001e418c-7e23-43d4-8264-d91e5eff2e94-proxy-tls" pod "001e418c-7e23-43d4-8264-d91e5eff2e94" (UID: "001e418c-7e23-43d4-8264-d91e5eff2e94"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:37.988980 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:37.988954    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-api-access-6g4qq" pod "001e418c-7e23-43d4-8264-d91e5eff2e94" (UID: "001e418c-7e23-43d4-8264-d91e5eff2e94"). InnerVolumeSpecName "kube-api-access-6g4qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:38.087247 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.087211    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g4qq\" (UniqueName: \"kubernetes.io/projected/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-api-access-6g4qq\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:48:38.087247 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.087242    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/001e418c-7e23-43d4-8264-d91e5eff2e94-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:48:38.087247 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.087252    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:48:38.087474 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.087263    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/001e418c-7e23-43d4-8264-d91e5eff2e94-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:48:38.087474 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.087272    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/001e418c-7e23-43d4-8264-d91e5eff2e94-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:48:38.195953 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.195924    4885 generic.go:356] "Generic (PLEG): container finished" podID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerID="e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e" exitCode=0
Oct 30 03:48:38.196128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.195967    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" event={"ID":"001e418c-7e23-43d4-8264-d91e5eff2e94","Type":"ContainerDied","Data":"e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e"}
Oct 30 03:48:38.196128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.195985    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"
Oct 30 03:48:38.196128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.196017    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7" event={"ID":"001e418c-7e23-43d4-8264-d91e5eff2e94","Type":"ContainerDied","Data":"0f082912c7afa8e537f5a2402b1c60019cbacb49213a099955cf5f3a2b967f29"}
Oct 30 03:48:38.196128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.196042    4885 scope.go:122] "RemoveContainer" containerID="b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134"
Oct 30 03:48:38.205123 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.205107    4885 scope.go:122] "RemoveContainer" containerID="e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e"
Oct 30 03:48:38.213034 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.213020    4885 scope.go:122] "RemoveContainer" containerID="bb05491321f3135fc02eeb30389ddb37624a8793f4a558d0570b5ea71a6516ae"
Oct 30 03:48:38.220457 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.220438    4885 scope.go:122] "RemoveContainer" containerID="b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134"
Oct 30 03:48:38.220758 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:48:38.220737    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134\": container with ID starting with b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134 not found: ID does not exist" containerID="b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134"
Oct 30 03:48:38.220831 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.220766    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134"} err="failed to get container status \"b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134\": rpc error: code = NotFound desc = could not find container \"b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134\": container with ID starting with b0bf56759765a58976012c8fbf2832bd99db1ed98a339144b232d36559798134 not found: ID does not exist"
Oct 30 03:48:38.220831 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.220782    4885 scope.go:122] "RemoveContainer" containerID="e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e"
Oct 30 03:48:38.221020 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:48:38.221006    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e\": container with ID starting with e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e not found: ID does not exist" containerID="e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e"
Oct 30 03:48:38.221067 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.221022    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e"} err="failed to get container status \"e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e\": rpc error: code = NotFound desc = could not find container \"e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e\": container with ID starting with e4bcfa8575077376b337011d4b17c1edd75b5cc0d4bd38de04ec10f253fa9b3e not found: ID does not exist"
Oct 30 03:48:38.221067 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.221034    4885 scope.go:122] "RemoveContainer" containerID="bb05491321f3135fc02eeb30389ddb37624a8793f4a558d0570b5ea71a6516ae"
Oct 30 03:48:38.221275 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:48:38.221256    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb05491321f3135fc02eeb30389ddb37624a8793f4a558d0570b5ea71a6516ae\": container with ID starting with bb05491321f3135fc02eeb30389ddb37624a8793f4a558d0570b5ea71a6516ae not found: ID does not exist" containerID="bb05491321f3135fc02eeb30389ddb37624a8793f4a558d0570b5ea71a6516ae"
Oct 30 03:48:38.221334 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.221282    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb05491321f3135fc02eeb30389ddb37624a8793f4a558d0570b5ea71a6516ae"} err="failed to get container status \"bb05491321f3135fc02eeb30389ddb37624a8793f4a558d0570b5ea71a6516ae\": rpc error: code = NotFound desc = could not find container \"bb05491321f3135fc02eeb30389ddb37624a8793f4a558d0570b5ea71a6516ae\": container with ID starting with bb05491321f3135fc02eeb30389ddb37624a8793f4a558d0570b5ea71a6516ae not found: ID does not exist"
Oct 30 03:48:38.242723 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.242666    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"]
Oct 30 03:48:38.252448 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.252428    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-thcj7"]
Oct 30 03:48:38.624981 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:38.624912    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" path="/var/lib/kubelet/pods/001e418c-7e23-43d4-8264-d91e5eff2e94/volumes"
Oct 30 03:48:43.935400 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:43.935363    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:48:52.646455 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:52.646421    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"]
Oct 30 03:48:52.646981 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:52.646745    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container" containerID="cri-o://c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3" gracePeriod=30
Oct 30 03:48:52.646981 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:52.646765    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kube-rbac-proxy" containerID="cri-o://3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46" gracePeriod=30
Oct 30 03:48:53.260329 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:53.260298    4885 generic.go:356] "Generic (PLEG): container finished" podID="c104160b-0599-4481-ab57-86c3a25edaca" containerID="3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46" exitCode=0
Oct 30 03:48:53.260524 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:53.260378    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" event={"ID":"c104160b-0599-4481-ab57-86c3a25edaca","Type":"ContainerDied","Data":"3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46"}
Oct 30 03:48:53.929931 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:53.929885    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.84:8643/healthz\": dial tcp 10.132.0.84:8643: connect: connection refused"
Oct 30 03:48:53.935297 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:53.935273    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.84:8080: connect: connection refused"
Oct 30 03:48:55.922043 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:48:55.922011    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:48:56.699866 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.699837    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:48:56.730670 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.730645    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c104160b-0599-4481-ab57-86c3a25edaca-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c104160b-0599-4481-ab57-86c3a25edaca-kserve-provision-location\") pod \"c104160b-0599-4481-ab57-86c3a25edaca\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") "
Oct 30 03:48:56.730808 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.730689    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c104160b-0599-4481-ab57-86c3a25edaca-kube-api-access-ww7sl\" (UniqueName: \"kubernetes.io/projected/c104160b-0599-4481-ab57-86c3a25edaca-kube-api-access-ww7sl\") pod \"c104160b-0599-4481-ab57-86c3a25edaca\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") "
Oct 30 03:48:56.730808 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.730711    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-kube-rbac-proxy-sar-config\") pod \"c104160b-0599-4481-ab57-86c3a25edaca\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") "
Oct 30 03:48:56.731005 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.730977    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c104160b-0599-4481-ab57-86c3a25edaca-kserve-provision-location" pod "c104160b-0599-4481-ab57-86c3a25edaca" (UID: "c104160b-0599-4481-ab57-86c3a25edaca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:48:56.731081 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.731048    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-kube-rbac-proxy-sar-config" pod "c104160b-0599-4481-ab57-86c3a25edaca" (UID: "c104160b-0599-4481-ab57-86c3a25edaca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:56.732870 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.732850    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c104160b-0599-4481-ab57-86c3a25edaca-kube-api-access-ww7sl" pod "c104160b-0599-4481-ab57-86c3a25edaca" (UID: "c104160b-0599-4481-ab57-86c3a25edaca"). InnerVolumeSpecName "kube-api-access-ww7sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:56.831335 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.831264    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-cabundle-cert\") pod \"c104160b-0599-4481-ab57-86c3a25edaca\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") "
Oct 30 03:48:56.831335 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.831306    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c104160b-0599-4481-ab57-86c3a25edaca-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c104160b-0599-4481-ab57-86c3a25edaca-proxy-tls\") pod \"c104160b-0599-4481-ab57-86c3a25edaca\" (UID: \"c104160b-0599-4481-ab57-86c3a25edaca\") "
Oct 30 03:48:56.831534 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.831476    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c104160b-0599-4481-ab57-86c3a25edaca-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:48:56.831534 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.831494    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww7sl\" (UniqueName: \"kubernetes.io/projected/c104160b-0599-4481-ab57-86c3a25edaca-kube-api-access-ww7sl\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:48:56.831534 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.831509    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:48:56.831668 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.831629    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-cabundle-cert" pod "c104160b-0599-4481-ab57-86c3a25edaca" (UID: "c104160b-0599-4481-ab57-86c3a25edaca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:56.833415 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.833392    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c104160b-0599-4481-ab57-86c3a25edaca-proxy-tls" pod "c104160b-0599-4481-ab57-86c3a25edaca" (UID: "c104160b-0599-4481-ab57-86c3a25edaca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:56.931934 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.931903    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c104160b-0599-4481-ab57-86c3a25edaca-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:48:56.931934 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:56.931930    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c104160b-0599-4481-ab57-86c3a25edaca-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:48:57.281634 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.281581    4885 generic.go:356] "Generic (PLEG): container finished" podID="c104160b-0599-4481-ab57-86c3a25edaca" containerID="c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3" exitCode=0
Oct 30 03:48:57.281802 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.281650    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"
Oct 30 03:48:57.281802 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.281649    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" event={"ID":"c104160b-0599-4481-ab57-86c3a25edaca","Type":"ContainerDied","Data":"c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3"}
Oct 30 03:48:57.281802 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.281717    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd" event={"ID":"c104160b-0599-4481-ab57-86c3a25edaca","Type":"ContainerDied","Data":"e928d2b93f295c38194b9ad3d90fb7073443a3db08d53eca8cdc32d3fce11325"}
Oct 30 03:48:57.281802 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.281745    4885 scope.go:122] "RemoveContainer" containerID="3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46"
Oct 30 03:48:57.290186 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.290169    4885 scope.go:122] "RemoveContainer" containerID="c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3"
Oct 30 03:48:57.297647 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.297626    4885 scope.go:122] "RemoveContainer" containerID="df63f78ac0af92be4b5b2e50cfc92f3d0759a08ae3ccfabec80261b3437279fa"
Oct 30 03:48:57.304816 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.304797    4885 scope.go:122] "RemoveContainer" containerID="3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46"
Oct 30 03:48:57.305097 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:48:57.305077    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46\": container with ID starting with 3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46 not found: ID does not exist" containerID="3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46"
Oct 30 03:48:57.305161 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.305107    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46"} err="failed to get container status \"3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46\": rpc error: code = NotFound desc = could not find container \"3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46\": container with ID starting with 3e7abcf29a4c0e41366846f1a2d68596e7bf87985ec3c095a122268efbe69d46 not found: ID does not exist"
Oct 30 03:48:57.305161 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.305124    4885 scope.go:122] "RemoveContainer" containerID="c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3"
Oct 30 03:48:57.305367 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:48:57.305350    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3\": container with ID starting with c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3 not found: ID does not exist" containerID="c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3"
Oct 30 03:48:57.305408 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.305377    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3"} err="failed to get container status \"c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3\": rpc error: code = NotFound desc = could not find container \"c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3\": container with ID starting with c71734d3656a00a235d3d15c1e6ac764cab8e42816d93272d28c1e6325f836f3 not found: ID does not exist"
Oct 30 03:48:57.305408 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.305393    4885 scope.go:122] "RemoveContainer" containerID="df63f78ac0af92be4b5b2e50cfc92f3d0759a08ae3ccfabec80261b3437279fa"
Oct 30 03:48:57.305608 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:48:57.305594    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df63f78ac0af92be4b5b2e50cfc92f3d0759a08ae3ccfabec80261b3437279fa\": container with ID starting with df63f78ac0af92be4b5b2e50cfc92f3d0759a08ae3ccfabec80261b3437279fa not found: ID does not exist" containerID="df63f78ac0af92be4b5b2e50cfc92f3d0759a08ae3ccfabec80261b3437279fa"
Oct 30 03:48:57.305674 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.305633    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df63f78ac0af92be4b5b2e50cfc92f3d0759a08ae3ccfabec80261b3437279fa"} err="failed to get container status \"df63f78ac0af92be4b5b2e50cfc92f3d0759a08ae3ccfabec80261b3437279fa\": rpc error: code = NotFound desc = could not find container \"df63f78ac0af92be4b5b2e50cfc92f3d0759a08ae3ccfabec80261b3437279fa\": container with ID starting with df63f78ac0af92be4b5b2e50cfc92f3d0759a08ae3ccfabec80261b3437279fa not found: ID does not exist"
Oct 30 03:48:57.337666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.337646    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"]
Oct 30 03:48:57.347233 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:57.347212    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-runtime-predictor-68fd78d655-td9bd"]
Oct 30 03:48:58.204608 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.204574    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"]
Oct 30 03:48:58.205018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.204915    4885 cpu_manager.go:414] "containerMap: removing container" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container"
Oct 30 03:48:58.205018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.204928    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container"
Oct 30 03:48:58.205018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.204968    4885 cpu_manager.go:414] "containerMap: removing container" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kube-rbac-proxy"
Oct 30 03:48:58.205018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.204977    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kube-rbac-proxy"
Oct 30 03:48:58.205018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.204995    4885 cpu_manager.go:414] "containerMap: removing container" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="storage-initializer"
Oct 30 03:48:58.205018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.205003    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="storage-initializer"
Oct 30 03:48:58.205018 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.205016    4885 cpu_manager.go:414] "containerMap: removing container" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="storage-initializer"
Oct 30 03:48:58.205378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.205025    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="storage-initializer"
Oct 30 03:48:58.205378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.205040    4885 cpu_manager.go:414] "containerMap: removing container" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container"
Oct 30 03:48:58.205378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.205048    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container"
Oct 30 03:48:58.205378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.205071    4885 cpu_manager.go:414] "containerMap: removing container" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kube-rbac-proxy"
Oct 30 03:48:58.205378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.205079    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kube-rbac-proxy"
Oct 30 03:48:58.205378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.205165    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kserve-container"
Oct 30 03:48:58.205378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.205176    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kube-rbac-proxy"
Oct 30 03:48:58.205378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.205186    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c104160b-0599-4481-ab57-86c3a25edaca" containerName="kserve-container"
Oct 30 03:48:58.205378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.205197    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="001e418c-7e23-43d4-8264-d91e5eff2e94" containerName="kube-rbac-proxy"
Oct 30 03:48:58.208397 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.208377    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.221429 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.221404    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:48:58.231410 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.231370    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 03:48:58.231410 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.231379    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:48:58.231410 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.231395    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:48:58.231720 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.231424    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:48:58.231720 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.231430    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Oct 30 03:48:58.231720 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.231527    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:48:58.233220 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.233203    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"]
Oct 30 03:48:58.242301 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.242275    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e910052-f820-4895-b6a3-ff6fe678f277-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.242407 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.242333    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znnvp\" (UniqueName: \"kubernetes.io/projected/8e910052-f820-4895-b6a3-ff6fe678f277-kube-api-access-znnvp\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.242449 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.242413    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e910052-f820-4895-b6a3-ff6fe678f277-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.242495 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.242454    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.242495 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.242485    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.343366 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.343339    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znnvp\" (UniqueName: \"kubernetes.io/projected/8e910052-f820-4895-b6a3-ff6fe678f277-kube-api-access-znnvp\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.343521 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.343374    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e910052-f820-4895-b6a3-ff6fe678f277-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.343521 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.343436    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.343521 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.343480    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.343680 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.343538    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e910052-f820-4895-b6a3-ff6fe678f277-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.343905 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.343881    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e910052-f820-4895-b6a3-ff6fe678f277-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.344254 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.344238    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.344305 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.344259    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.346230 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.346211    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e910052-f820-4895-b6a3-ff6fe678f277-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.364596 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.364577    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znnvp\" (UniqueName: \"kubernetes.io/projected/8e910052-f820-4895-b6a3-ff6fe678f277-kube-api-access-znnvp\") pod \"isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.525541 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.525457    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:48:58.624451 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.624419    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c104160b-0599-4481-ab57-86c3a25edaca" path="/var/lib/kubelet/pods/c104160b-0599-4481-ab57-86c3a25edaca/volumes"
Oct 30 03:48:58.831843 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:58.831760    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"]
Oct 30 03:48:58.835133 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:48:58.835102    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e910052_f820_4895_b6a3_ff6fe678f277.slice/crio-19dde266fcbdc7d4ca390d06f28f382a27ef35190f30cf0d65f5c2ca4742ed17 WatchSource:0}: Error finding container 19dde266fcbdc7d4ca390d06f28f382a27ef35190f30cf0d65f5c2ca4742ed17: Status 404 returned error can't find the container with id 19dde266fcbdc7d4ca390d06f28f382a27ef35190f30cf0d65f5c2ca4742ed17
Oct 30 03:48:59.293375 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:59.293340    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" event={"ID":"8e910052-f820-4895-b6a3-ff6fe678f277","Type":"ContainerStarted","Data":"f6f92ffb3964794abd83666d141e21d573c0dfe14a50c4ed5b79b537dbb34813"}
Oct 30 03:48:59.293889 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:48:59.293409    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" event={"ID":"8e910052-f820-4895-b6a3-ff6fe678f277","Type":"ContainerStarted","Data":"19dde266fcbdc7d4ca390d06f28f382a27ef35190f30cf0d65f5c2ca4742ed17"}
Oct 30 03:49:03.316385 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:03.316351    4885 generic.go:356] "Generic (PLEG): container finished" podID="8e910052-f820-4895-b6a3-ff6fe678f277" containerID="f6f92ffb3964794abd83666d141e21d573c0dfe14a50c4ed5b79b537dbb34813" exitCode=0
Oct 30 03:49:03.316776 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:03.316424    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" event={"ID":"8e910052-f820-4895-b6a3-ff6fe678f277","Type":"ContainerDied","Data":"f6f92ffb3964794abd83666d141e21d573c0dfe14a50c4ed5b79b537dbb34813"}
Oct 30 03:49:04.322549 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:04.322517    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" event={"ID":"8e910052-f820-4895-b6a3-ff6fe678f277","Type":"ContainerStarted","Data":"19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e"}
Oct 30 03:49:04.323033 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:04.322577    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" event={"ID":"8e910052-f820-4895-b6a3-ff6fe678f277","Type":"ContainerStarted","Data":"b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82"}
Oct 30 03:49:04.323033 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:04.322833    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:49:04.383587 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:04.383545    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" podStartSLOduration=2.752435518 podStartE2EDuration="6.383532191s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.631096673s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:04.373152443 +0000 UTC m=+3249.883321482" watchObservedRunningTime="2026-10-30 03:49:04.383532191 +0000 UTC m=+3249.893701252"
Oct 30 03:49:05.327399 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:05.327372    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:49:11.337273 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:11.337246    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:49:25.921883 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:49:25.921846    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:49:41.433653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:41.433539    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Oct 30 03:49:51.339747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:51.339718    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:49:54.951225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:54.951101    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:49:54.957088 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:54.957064    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:49:54.957357 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:54.957340    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:49:54.962510 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:54.962489    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:49:55.921649 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:49:55.921599    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:49:58.519949 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:58.519916    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"]
Oct 30 03:49:58.520393 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:58.520193    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" containerName="kserve-container" containerID="cri-o://b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82" gracePeriod=30
Oct 30 03:49:58.520393 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:58.520257    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" containerName="kube-rbac-proxy" containerID="cri-o://19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e" gracePeriod=30
Oct 30 03:49:59.552243 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:59.552213    4885 generic.go:356] "Generic (PLEG): container finished" podID="8e910052-f820-4895-b6a3-ff6fe678f277" containerID="19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e" exitCode=0
Oct 30 03:49:59.552653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:49:59.552255    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" event={"ID":"8e910052-f820-4895-b6a3-ff6fe678f277","Type":"ContainerDied","Data":"19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e"}
Oct 30 03:50:01.332947 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:01.332900    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.85:8643/healthz\": dial tcp 10.132.0.85:8643: connect: connection refused"
Oct 30 03:50:05.760259 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.760234    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:50:05.783766 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.783739    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e910052-f820-4895-b6a3-ff6fe678f277-kube-api-access-znnvp\" (UniqueName: \"kubernetes.io/projected/8e910052-f820-4895-b6a3-ff6fe678f277-kube-api-access-znnvp\") pod \"8e910052-f820-4895-b6a3-ff6fe678f277\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") "
Oct 30 03:50:05.783908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.783790    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-kube-rbac-proxy-sar-config\") pod \"8e910052-f820-4895-b6a3-ff6fe678f277\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") "
Oct 30 03:50:05.783908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.783851    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e910052-f820-4895-b6a3-ff6fe678f277-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e910052-f820-4895-b6a3-ff6fe678f277-kserve-provision-location\") pod \"8e910052-f820-4895-b6a3-ff6fe678f277\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") "
Oct 30 03:50:05.783908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.783883    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-cabundle-cert\") pod \"8e910052-f820-4895-b6a3-ff6fe678f277\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") "
Oct 30 03:50:05.784089 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.783925    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e910052-f820-4895-b6a3-ff6fe678f277-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e910052-f820-4895-b6a3-ff6fe678f277-proxy-tls\") pod \"8e910052-f820-4895-b6a3-ff6fe678f277\" (UID: \"8e910052-f820-4895-b6a3-ff6fe678f277\") "
Oct 30 03:50:05.784196 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.784166    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e910052-f820-4895-b6a3-ff6fe678f277-kserve-provision-location" pod "8e910052-f820-4895-b6a3-ff6fe678f277" (UID: "8e910052-f820-4895-b6a3-ff6fe678f277"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:05.784273 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.784245    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-cabundle-cert" pod "8e910052-f820-4895-b6a3-ff6fe678f277" (UID: "8e910052-f820-4895-b6a3-ff6fe678f277"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:05.784319 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.784253    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-kube-rbac-proxy-sar-config" pod "8e910052-f820-4895-b6a3-ff6fe678f277" (UID: "8e910052-f820-4895-b6a3-ff6fe678f277"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:05.785988 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.785965    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e910052-f820-4895-b6a3-ff6fe678f277-kube-api-access-znnvp" pod "8e910052-f820-4895-b6a3-ff6fe678f277" (UID: "8e910052-f820-4895-b6a3-ff6fe678f277"). InnerVolumeSpecName "kube-api-access-znnvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:05.786089 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.786039    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e910052-f820-4895-b6a3-ff6fe678f277-proxy-tls" pod "8e910052-f820-4895-b6a3-ff6fe678f277" (UID: "8e910052-f820-4895-b6a3-ff6fe678f277"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:05.884771 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.884735    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e910052-f820-4895-b6a3-ff6fe678f277-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:50:05.884771 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.884762    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:50:05.884771 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.884774    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e910052-f820-4895-b6a3-ff6fe678f277-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:50:05.884991 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.884782    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znnvp\" (UniqueName: \"kubernetes.io/projected/8e910052-f820-4895-b6a3-ff6fe678f277-kube-api-access-znnvp\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:50:05.884991 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:05.884791    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e910052-f820-4895-b6a3-ff6fe678f277-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:50:06.583528 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.583494    4885 generic.go:356] "Generic (PLEG): container finished" podID="8e910052-f820-4895-b6a3-ff6fe678f277" containerID="b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82" exitCode=0
Oct 30 03:50:06.583729 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.583554    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"
Oct 30 03:50:06.583729 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.583573    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" event={"ID":"8e910052-f820-4895-b6a3-ff6fe678f277","Type":"ContainerDied","Data":"b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82"}
Oct 30 03:50:06.583729 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.583667    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt" event={"ID":"8e910052-f820-4895-b6a3-ff6fe678f277","Type":"ContainerDied","Data":"19dde266fcbdc7d4ca390d06f28f382a27ef35190f30cf0d65f5c2ca4742ed17"}
Oct 30 03:50:06.583729 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.583705    4885 scope.go:122] "RemoveContainer" containerID="19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e"
Oct 30 03:50:06.592821 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.592799    4885 scope.go:122] "RemoveContainer" containerID="b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82"
Oct 30 03:50:06.600818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.600802    4885 scope.go:122] "RemoveContainer" containerID="f6f92ffb3964794abd83666d141e21d573c0dfe14a50c4ed5b79b537dbb34813"
Oct 30 03:50:06.608395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.608377    4885 scope.go:122] "RemoveContainer" containerID="19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e"
Oct 30 03:50:06.608662 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:50:06.608641    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e\": container with ID starting with 19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e not found: ID does not exist" containerID="19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e"
Oct 30 03:50:06.608806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.608673    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e"} err="failed to get container status \"19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e\": rpc error: code = NotFound desc = could not find container \"19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e\": container with ID starting with 19f6869ef902329803ec46560c2a788ffd1e010ce89c44d9cd7988a27b68e05e not found: ID does not exist"
Oct 30 03:50:06.608806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.608696    4885 scope.go:122] "RemoveContainer" containerID="b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82"
Oct 30 03:50:06.608927 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:50:06.608910    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82\": container with ID starting with b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82 not found: ID does not exist" containerID="b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82"
Oct 30 03:50:06.608970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.608933    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82"} err="failed to get container status \"b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82\": rpc error: code = NotFound desc = could not find container \"b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82\": container with ID starting with b47dd36dc0f4f9f9c9b528074ba2a26d1458d5bedcc95ad88658a88e28acea82 not found: ID does not exist"
Oct 30 03:50:06.608970 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.608950    4885 scope.go:122] "RemoveContainer" containerID="f6f92ffb3964794abd83666d141e21d573c0dfe14a50c4ed5b79b537dbb34813"
Oct 30 03:50:06.609210 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:50:06.609196    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f92ffb3964794abd83666d141e21d573c0dfe14a50c4ed5b79b537dbb34813\": container with ID starting with f6f92ffb3964794abd83666d141e21d573c0dfe14a50c4ed5b79b537dbb34813 not found: ID does not exist" containerID="f6f92ffb3964794abd83666d141e21d573c0dfe14a50c4ed5b79b537dbb34813"
Oct 30 03:50:06.609255 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.609214    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f92ffb3964794abd83666d141e21d573c0dfe14a50c4ed5b79b537dbb34813"} err="failed to get container status \"f6f92ffb3964794abd83666d141e21d573c0dfe14a50c4ed5b79b537dbb34813\": rpc error: code = NotFound desc = could not find container \"f6f92ffb3964794abd83666d141e21d573c0dfe14a50c4ed5b79b537dbb34813\": container with ID starting with f6f92ffb3964794abd83666d141e21d573c0dfe14a50c4ed5b79b537dbb34813 not found: ID does not exist"
Oct 30 03:50:06.639307 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.639280    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"]
Oct 30 03:50:06.649407 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:06.649379    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-runtime-predictor-7fb4bddbd4-9fggt"]
Oct 30 03:50:08.102091 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.102061    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"]
Oct 30 03:50:08.102560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.102388    4885 cpu_manager.go:414] "containerMap: removing container" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" containerName="kserve-container"
Oct 30 03:50:08.102560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.102406    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" containerName="kserve-container"
Oct 30 03:50:08.102560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.102418    4885 cpu_manager.go:414] "containerMap: removing container" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" containerName="kube-rbac-proxy"
Oct 30 03:50:08.102560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.102425    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" containerName="kube-rbac-proxy"
Oct 30 03:50:08.102560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.102449    4885 cpu_manager.go:414] "containerMap: removing container" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" containerName="storage-initializer"
Oct 30 03:50:08.102560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.102457    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" containerName="storage-initializer"
Oct 30 03:50:08.102560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.102536    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" containerName="kserve-container"
Oct 30 03:50:08.102560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.102544    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" containerName="kube-rbac-proxy"
Oct 30 03:50:08.107207 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.107188    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.131843 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.131813    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:50:08.131954 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.131867    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:50:08.131954 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.131883    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:50:08.131954 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.131914    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:50:08.131954 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.131939    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Oct 30 03:50:08.132122 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.132000    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:50:08.132243 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.132223    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 30 03:50:08.133441 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.133426    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"]
Oct 30 03:50:08.198758 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.198734    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.198874 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.198765    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.198874 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.198785    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qffjx\" (UniqueName: \"kubernetes.io/projected/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-api-access-qffjx\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.198874 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.198800    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.198974 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.198951    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc80809-cdbc-45e2-ab60-fc9b96953c90-proxy-tls\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.299789 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.299759    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc80809-cdbc-45e2-ab60-fc9b96953c90-proxy-tls\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.299913 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.299823    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.299913 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.299844    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.299913 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.299863    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qffjx\" (UniqueName: \"kubernetes.io/projected/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-api-access-qffjx\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.299913 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.299881    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.300367 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.300343    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.300553 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.300504    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.300765 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.300561    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.302354 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.302332    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc80809-cdbc-45e2-ab60-fc9b96953c90-proxy-tls\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.321017 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.320989    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qffjx\" (UniqueName: \"kubernetes.io/projected/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-api-access-qffjx\") pod \"isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") " pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.425169 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.425096    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:08.624092 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.624063    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e910052-f820-4895-b6a3-ff6fe678f277" path="/var/lib/kubelet/pods/8e910052-f820-4895-b6a3-ff6fe678f277/volumes"
Oct 30 03:50:08.728747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:08.728725    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"]
Oct 30 03:50:08.731358 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:50:08.731334    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc80809_cdbc_45e2_ab60_fc9b96953c90.slice/crio-322e8061d7c66db76ad25bb26dc695af33020eab3b278ea6f92582a33dd86786 WatchSource:0}: Error finding container 322e8061d7c66db76ad25bb26dc695af33020eab3b278ea6f92582a33dd86786: Status 404 returned error can't find the container with id 322e8061d7c66db76ad25bb26dc695af33020eab3b278ea6f92582a33dd86786
Oct 30 03:50:09.598630 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:09.598581    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" event={"ID":"1dc80809-cdbc-45e2-ab60-fc9b96953c90","Type":"ContainerStarted","Data":"416706f866f47510b5b37bb48f5b90fb47a932b0daef9649a75e621c2452f264"}
Oct 30 03:50:09.599212 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:09.598656    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" event={"ID":"1dc80809-cdbc-45e2-ab60-fc9b96953c90","Type":"ContainerStarted","Data":"322e8061d7c66db76ad25bb26dc695af33020eab3b278ea6f92582a33dd86786"}
Oct 30 03:50:13.619016 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:13.618982    4885 generic.go:356] "Generic (PLEG): container finished" podID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerID="416706f866f47510b5b37bb48f5b90fb47a932b0daef9649a75e621c2452f264" exitCode=0
Oct 30 03:50:13.619392 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:13.619053    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" event={"ID":"1dc80809-cdbc-45e2-ab60-fc9b96953c90","Type":"ContainerDied","Data":"416706f866f47510b5b37bb48f5b90fb47a932b0daef9649a75e621c2452f264"}
Oct 30 03:50:14.626222 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:14.626189    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" event={"ID":"1dc80809-cdbc-45e2-ab60-fc9b96953c90","Type":"ContainerStarted","Data":"e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5"}
Oct 30 03:50:14.626587 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:14.626246    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" event={"ID":"1dc80809-cdbc-45e2-ab60-fc9b96953c90","Type":"ContainerStarted","Data":"664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972"}
Oct 30 03:50:14.626587 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:14.626427    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:14.687183 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:14.687135    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podStartSLOduration=2.802597814 podStartE2EDuration="6.687123765s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.884525951s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:14.67696162 +0000 UTC m=+3320.187130646" watchObservedRunningTime="2026-10-30 03:50:14.687123765 +0000 UTC m=+3320.197292790"
Oct 30 03:50:15.630585 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:15.630549    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:15.631935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:15.631894    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 03:50:16.634560 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:16.634517    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 03:50:21.639214 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:21.639184    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:50:21.639845 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:21.639819    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 03:50:25.922321 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:50:25.922286    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:50:31.639849 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:31.639798    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 03:50:41.639976 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:41.639935    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 03:50:51.639845 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:50:51.639795    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 03:50:55.922149 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:50:55.922111    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:51:01.640513 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:01.640465    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 03:51:11.640489 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:11.640409    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 03:51:21.640816 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:21.640789    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:51:25.921706 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:51:25.921670    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:51:28.941371 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:28.941339    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"]
Oct 30 03:51:28.941901 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:28.941596    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container" containerID="cri-o://664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972" gracePeriod=30
Oct 30 03:51:28.941901 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:28.941646    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kube-rbac-proxy" containerID="cri-o://e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5" gracePeriod=30
Oct 30 03:51:29.943778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:29.943698    4885 generic.go:356] "Generic (PLEG): container finished" podID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerID="e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5" exitCode=0
Oct 30 03:51:29.943778 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:29.943736    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" event={"ID":"1dc80809-cdbc-45e2-ab60-fc9b96953c90","Type":"ContainerDied","Data":"e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5"}
Oct 30 03:51:31.635675 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:31.635633    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.86:8643/healthz\": dial tcp 10.132.0.86:8643: connect: connection refused"
Oct 30 03:51:31.639975 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:31.639942    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 30 03:51:32.779540 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.779516    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:51:32.873453 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.873421    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-cabundle-cert\") pod \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") "
Oct 30 03:51:32.873608 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.873494    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-api-access-qffjx\" (UniqueName: \"kubernetes.io/projected/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-api-access-qffjx\") pod \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") "
Oct 30 03:51:32.873608 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.873568    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dc80809-cdbc-45e2-ab60-fc9b96953c90-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc80809-cdbc-45e2-ab60-fc9b96953c90-proxy-tls\") pod \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") "
Oct 30 03:51:32.873765 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.873638    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kserve-provision-location\") pod \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") "
Oct 30 03:51:32.873765 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.873667    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-rbac-proxy-sar-config\") pod \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\" (UID: \"1dc80809-cdbc-45e2-ab60-fc9b96953c90\") "
Oct 30 03:51:32.873926 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.873900    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-cabundle-cert" pod "1dc80809-cdbc-45e2-ab60-fc9b96953c90" (UID: "1dc80809-cdbc-45e2-ab60-fc9b96953c90"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:51:32.873995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.873941    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kserve-provision-location" pod "1dc80809-cdbc-45e2-ab60-fc9b96953c90" (UID: "1dc80809-cdbc-45e2-ab60-fc9b96953c90"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:51:32.874046 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.874001    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-rbac-proxy-sar-config" pod "1dc80809-cdbc-45e2-ab60-fc9b96953c90" (UID: "1dc80809-cdbc-45e2-ab60-fc9b96953c90"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:51:32.875831 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.875802    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-api-access-qffjx" pod "1dc80809-cdbc-45e2-ab60-fc9b96953c90" (UID: "1dc80809-cdbc-45e2-ab60-fc9b96953c90"). InnerVolumeSpecName "kube-api-access-qffjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:51:32.875894 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.875866    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc80809-cdbc-45e2-ab60-fc9b96953c90-proxy-tls" pod "1dc80809-cdbc-45e2-ab60-fc9b96953c90" (UID: "1dc80809-cdbc-45e2-ab60-fc9b96953c90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:51:32.958684 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.958662    4885 generic.go:356] "Generic (PLEG): container finished" podID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerID="664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972" exitCode=0
Oct 30 03:51:32.958802 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.958693    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" event={"ID":"1dc80809-cdbc-45e2-ab60-fc9b96953c90","Type":"ContainerDied","Data":"664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972"}
Oct 30 03:51:32.958802 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.958733    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"
Oct 30 03:51:32.958802 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.958766    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7" event={"ID":"1dc80809-cdbc-45e2-ab60-fc9b96953c90","Type":"ContainerDied","Data":"322e8061d7c66db76ad25bb26dc695af33020eab3b278ea6f92582a33dd86786"}
Oct 30 03:51:32.958802 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.958799    4885 scope.go:122] "RemoveContainer" containerID="e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5"
Oct 30 03:51:32.967140 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.967125    4885 scope.go:122] "RemoveContainer" containerID="664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972"
Oct 30 03:51:32.974770 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.974740    4885 scope.go:122] "RemoveContainer" containerID="416706f866f47510b5b37bb48f5b90fb47a932b0daef9649a75e621c2452f264"
Oct 30 03:51:32.975011 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.974993    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc80809-cdbc-45e2-ab60-fc9b96953c90-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:51:32.975080 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.975017    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:51:32.975080 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.975032    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:51:32.975080 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.975045    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc80809-cdbc-45e2-ab60-fc9b96953c90-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:51:32.975080 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.975055    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qffjx\" (UniqueName: \"kubernetes.io/projected/1dc80809-cdbc-45e2-ab60-fc9b96953c90-kube-api-access-qffjx\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:51:32.982036 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.982021    4885 scope.go:122] "RemoveContainer" containerID="e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5"
Oct 30 03:51:32.982299 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:51:32.982280    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5\": container with ID starting with e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5 not found: ID does not exist" containerID="e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5"
Oct 30 03:51:32.982348 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.982312    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5"} err="failed to get container status \"e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5\": rpc error: code = NotFound desc = could not find container \"e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5\": container with ID starting with e063030e0db16d77bf4162d2aa7e726c77112b08f8948e0747092b88cb4cbce5 not found: ID does not exist"
Oct 30 03:51:32.982348 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.982334    4885 scope.go:122] "RemoveContainer" containerID="664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972"
Oct 30 03:51:32.982579 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:51:32.982561    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972\": container with ID starting with 664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972 not found: ID does not exist" containerID="664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972"
Oct 30 03:51:32.982666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.982589    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972"} err="failed to get container status \"664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972\": rpc error: code = NotFound desc = could not find container \"664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972\": container with ID starting with 664213cc4cac4df5735eec5d49b8df9f887b219892284baed84931cdd2498972 not found: ID does not exist"
Oct 30 03:51:32.982666 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.982626    4885 scope.go:122] "RemoveContainer" containerID="416706f866f47510b5b37bb48f5b90fb47a932b0daef9649a75e621c2452f264"
Oct 30 03:51:32.982857 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:51:32.982837    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416706f866f47510b5b37bb48f5b90fb47a932b0daef9649a75e621c2452f264\": container with ID starting with 416706f866f47510b5b37bb48f5b90fb47a932b0daef9649a75e621c2452f264 not found: ID does not exist" containerID="416706f866f47510b5b37bb48f5b90fb47a932b0daef9649a75e621c2452f264"
Oct 30 03:51:32.982895 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:32.982863    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416706f866f47510b5b37bb48f5b90fb47a932b0daef9649a75e621c2452f264"} err="failed to get container status \"416706f866f47510b5b37bb48f5b90fb47a932b0daef9649a75e621c2452f264\": rpc error: code = NotFound desc = could not find container \"416706f866f47510b5b37bb48f5b90fb47a932b0daef9649a75e621c2452f264\": container with ID starting with 416706f866f47510b5b37bb48f5b90fb47a932b0daef9649a75e621c2452f264 not found: ID does not exist"
Oct 30 03:51:33.004597 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:33.004576    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"]
Oct 30 03:51:33.012254 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:33.012229    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-xgboost-v2-predictor-5f55c59f4c-bp9c7"]
Oct 30 03:51:34.624263 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.624232    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" path="/var/lib/kubelet/pods/1dc80809-cdbc-45e2-ab60-fc9b96953c90/volumes"
Oct 30 03:51:34.800521 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.800489    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"]
Oct 30 03:51:34.800924 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.800901    4885 cpu_manager.go:414] "containerMap: removing container" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="storage-initializer"
Oct 30 03:51:34.800924 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.800925    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="storage-initializer"
Oct 30 03:51:34.801150 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.800951    4885 cpu_manager.go:414] "containerMap: removing container" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container"
Oct 30 03:51:34.801150 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.800960    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container"
Oct 30 03:51:34.801150 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.800976    4885 cpu_manager.go:414] "containerMap: removing container" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kube-rbac-proxy"
Oct 30 03:51:34.801150 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.800981    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kube-rbac-proxy"
Oct 30 03:51:34.801150 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.801049    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kserve-container"
Oct 30 03:51:34.801150 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.801056    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc80809-cdbc-45e2-ab60-fc9b96953c90" containerName="kube-rbac-proxy"
Oct 30 03:51:34.806020 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.806003    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:34.824109 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.824089    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:51:34.830752 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.830729    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"storage-config\""
Oct 30 03:51:34.830880 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.830752    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Oct 30 03:51:34.830880 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.830776    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:51:34.830880 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.830801    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:51:34.830880 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.830871    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Oct 30 03:51:34.835838 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.835808    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"]
Oct 30 03:51:34.887561 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.887505    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-556bb4d79b-x2dmg\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:34.887561 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.887536    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-556bb4d79b-x2dmg\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:34.887561 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.887555    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mkts\" (UniqueName: \"kubernetes.io/projected/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-api-access-4mkts\") pod \"isvc-sklearn-s3-predictor-556bb4d79b-x2dmg\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:34.887797 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.887647    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-556bb4d79b-x2dmg\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:34.988714 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.988688    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-556bb4d79b-x2dmg\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:34.988843 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.988719    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-556bb4d79b-x2dmg\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:34.988843 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.988738    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mkts\" (UniqueName: \"kubernetes.io/projected/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-api-access-4mkts\") pod \"isvc-sklearn-s3-predictor-556bb4d79b-x2dmg\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:34.988843 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.988764    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-556bb4d79b-x2dmg\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:34.989100 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.989081    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-556bb4d79b-x2dmg\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:34.989405 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.989384    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-556bb4d79b-x2dmg\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:34.991387 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:34.991371    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-556bb4d79b-x2dmg\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:35.010158 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:35.010130    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mkts\" (UniqueName: \"kubernetes.io/projected/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-api-access-4mkts\") pod \"isvc-sklearn-s3-predictor-556bb4d79b-x2dmg\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:35.122162 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:35.122126    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:35.428275 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:35.428245    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"]
Oct 30 03:51:35.431012 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:51:35.430983    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ffed30_6bec_4bdf_a952_3ec9dd3dad2a.slice/crio-3eaf9be9f0270d11713a697353ae5df1737bc53881d8d1ee7253bcdeac692f64 WatchSource:0}: Error finding container 3eaf9be9f0270d11713a697353ae5df1737bc53881d8d1ee7253bcdeac692f64: Status 404 returned error can't find the container with id 3eaf9be9f0270d11713a697353ae5df1737bc53881d8d1ee7253bcdeac692f64
Oct 30 03:51:35.974643 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:35.974592    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" event={"ID":"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a","Type":"ContainerStarted","Data":"56a44f5236c5ccd122ccfd9590332e58e14e7e1a3c9ee66bf6f6e89401b74fcc"}
Oct 30 03:51:35.975033 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:35.974661    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" event={"ID":"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a","Type":"ContainerStarted","Data":"3eaf9be9f0270d11713a697353ae5df1737bc53881d8d1ee7253bcdeac692f64"}
Oct 30 03:51:36.980747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:36.980715    4885 generic.go:356] "Generic (PLEG): container finished" podID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerID="56a44f5236c5ccd122ccfd9590332e58e14e7e1a3c9ee66bf6f6e89401b74fcc" exitCode=0
Oct 30 03:51:36.981196 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:36.980774    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" event={"ID":"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a","Type":"ContainerDied","Data":"56a44f5236c5ccd122ccfd9590332e58e14e7e1a3c9ee66bf6f6e89401b74fcc"}
Oct 30 03:51:37.988370 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:37.988333    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" event={"ID":"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a","Type":"ContainerStarted","Data":"ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1"}
Oct 30 03:51:37.988818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:37.988389    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" event={"ID":"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a","Type":"ContainerStarted","Data":"024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737"}
Oct 30 03:51:37.988818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:37.988416    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:38.993084 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:38.993053    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:38.994378 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:38.994354    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 03:51:39.998040 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:39.997995    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 03:51:45.003120 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:45.003090    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:51:45.003580 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:45.003557    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 03:51:45.054988 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:45.054936    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podStartSLOduration=10.057021435 podStartE2EDuration="11.054916936s" totalImagesPullingTime="0s" totalInitContainerRuntime="997.895501ms" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:38.038220411 +0000 UTC m=+3403.548389448" watchObservedRunningTime="2026-10-30 03:51:45.054916936 +0000 UTC m=+3410.565085961"
Oct 30 03:51:55.004383 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:51:55.004346    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 03:51:55.921317 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:51:55.921282    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:52:05.003870 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:05.003830    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 03:52:15.004119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:15.004076    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 03:52:25.003952 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:25.003911    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 03:52:25.921196 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:52:25.921157    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:52:35.003903 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:35.003859    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 03:52:45.003748 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:45.003707    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 03:52:46.812450 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.812416    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"]
Oct 30 03:52:46.816174 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.816153    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:46.828852 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.828820    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:52:46.828935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.828881    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:52:46.838918 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.838890    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:52:46.839028 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.838945    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:52:46.839028 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.838944    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:52:46.839028 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.839016    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 03:52:46.839838 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.839797    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Oct 30 03:52:46.843416 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.843394    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"]
Oct 30 03:52:46.991948 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.991899    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7j8t\" (UniqueName: \"kubernetes.io/projected/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-api-access-t7j8t\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:46.992132 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.992042    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cee2545e-7520-4c0c-8c7b-123738f9177c-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:46.992132 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.992122    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:46.992231 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.992178    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cee2545e-7520-4c0c-8c7b-123738f9177c-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:46.992286 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:46.992272    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:47.093598 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:47.093518    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:47.093598 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:47.093554    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cee2545e-7520-4c0c-8c7b-123738f9177c-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:47.093598 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:47.093578    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:47.093895 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:47.093780    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7j8t\" (UniqueName: \"kubernetes.io/projected/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-api-access-t7j8t\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:47.093895 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:47.093837    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cee2545e-7520-4c0c-8c7b-123738f9177c-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:47.094176 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:47.094156    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cee2545e-7520-4c0c-8c7b-123738f9177c-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:47.094277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:47.094258    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:47.094320 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:47.094306    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:47.096195 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:47.096175    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cee2545e-7520-4c0c-8c7b-123738f9177c-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:47.116194 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:47.116168    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7j8t\" (UniqueName: \"kubernetes.io/projected/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-api-access-t7j8t\") pod \"isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:47.133174 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:47.133148    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:47.437978 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:47.437928    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"]
Oct 30 03:52:47.441119 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:52:47.441088    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee2545e_7520_4c0c_8c7b_123738f9177c.slice/crio-2a5d52b475591bc16b48c4b24cf9a6fe74b2b24480d6da784caba431cc634b42 WatchSource:0}: Error finding container 2a5d52b475591bc16b48c4b24cf9a6fe74b2b24480d6da784caba431cc634b42: Status 404 returned error can't find the container with id 2a5d52b475591bc16b48c4b24cf9a6fe74b2b24480d6da784caba431cc634b42
Oct 30 03:52:48.287868 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:48.287832    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2" event={"ID":"cee2545e-7520-4c0c-8c7b-123738f9177c","Type":"ContainerStarted","Data":"b5bc9be19757d496e9fb2b7b15337e3ef4535f0cf0f89fffaae6e0dcd0140166"}
Oct 30 03:52:48.288299 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:48.287890    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2" event={"ID":"cee2545e-7520-4c0c-8c7b-123738f9177c","Type":"ContainerStarted","Data":"2a5d52b475591bc16b48c4b24cf9a6fe74b2b24480d6da784caba431cc634b42"}
Oct 30 03:52:48.619718 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:48.619608    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 03:52:52.306907 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:52.306873    4885 generic.go:356] "Generic (PLEG): container finished" podID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerID="b5bc9be19757d496e9fb2b7b15337e3ef4535f0cf0f89fffaae6e0dcd0140166" exitCode=0
Oct 30 03:52:52.307400 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:52.306949    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2" event={"ID":"cee2545e-7520-4c0c-8c7b-123738f9177c","Type":"ContainerDied","Data":"b5bc9be19757d496e9fb2b7b15337e3ef4535f0cf0f89fffaae6e0dcd0140166"}
Oct 30 03:52:53.314045 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:53.314008    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2" event={"ID":"cee2545e-7520-4c0c-8c7b-123738f9177c","Type":"ContainerStarted","Data":"796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350"}
Oct 30 03:52:53.314437 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:53.314090    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2" event={"ID":"cee2545e-7520-4c0c-8c7b-123738f9177c","Type":"ContainerStarted","Data":"e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb"}
Oct 30 03:52:53.314437 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:53.314314    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:53.364388 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:53.364343    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2" podStartSLOduration=3.560052672 podStartE2EDuration="7.364328372s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.8042757s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:52:53.363899098 +0000 UTC m=+3478.874068149" watchObservedRunningTime="2026-10-30 03:52:53.364328372 +0000 UTC m=+3478.874497395"
Oct 30 03:52:54.317853 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:54.317827    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:52:55.922926 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:52:55.922891    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:52:58.624333 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:52:58.624306    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:53:00.327342 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:00.327313    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:53:05.449846 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:05.449814    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"]
Oct 30 03:53:05.450364 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:05.450108    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container" containerID="cri-o://024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737" gracePeriod=30
Oct 30 03:53:05.450364 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:05.450168    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kube-rbac-proxy" containerID="cri-o://ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1" gracePeriod=30
Oct 30 03:53:06.370653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:06.370595    4885 generic.go:356] "Generic (PLEG): container finished" podID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerID="ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1" exitCode=0
Oct 30 03:53:06.370843 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:06.370673    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" event={"ID":"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a","Type":"ContainerDied","Data":"ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1"}
Oct 30 03:53:08.620507 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:08.620463    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 30 03:53:09.999242 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:09.999202    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.87:8643/healthz\": dial tcp 10.132.0.87:8643: connect: connection refused"
Oct 30 03:53:10.193239 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.193212    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:53:10.279889 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.279804    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-api-access-4mkts\" (UniqueName: \"kubernetes.io/projected/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-api-access-4mkts\") pod \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") "
Oct 30 03:53:10.279889 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.279839    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-proxy-tls\") pod \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") "
Oct 30 03:53:10.279889 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.279884    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kserve-provision-location\") pod \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") "
Oct 30 03:53:10.280120 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.279937    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-rbac-proxy-sar-config\") pod \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\" (UID: \"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a\") "
Oct 30 03:53:10.280243 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.280222    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kserve-provision-location" pod "33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" (UID: "33ffed30-6bec-4bdf-a952-3ec9dd3dad2a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:10.280322 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.280300    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-rbac-proxy-sar-config" pod "33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" (UID: "33ffed30-6bec-4bdf-a952-3ec9dd3dad2a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:53:10.282090 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.282071    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-proxy-tls" pod "33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" (UID: "33ffed30-6bec-4bdf-a952-3ec9dd3dad2a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:53:10.282269 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.282254    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-api-access-4mkts" pod "33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" (UID: "33ffed30-6bec-4bdf-a952-3ec9dd3dad2a"). InnerVolumeSpecName "kube-api-access-4mkts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:10.381509 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.381470    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mkts\" (UniqueName: \"kubernetes.io/projected/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-api-access-4mkts\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:53:10.381509 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.381504    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:53:10.381509 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.381517    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:53:10.381777 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.381529    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:53:10.388876 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.388849    4885 generic.go:356] "Generic (PLEG): container finished" podID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerID="024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737" exitCode=0
Oct 30 03:53:10.389045 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.388905    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"
Oct 30 03:53:10.389045 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.388941    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" event={"ID":"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a","Type":"ContainerDied","Data":"024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737"}
Oct 30 03:53:10.389045 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.389014    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg" event={"ID":"33ffed30-6bec-4bdf-a952-3ec9dd3dad2a","Type":"ContainerDied","Data":"3eaf9be9f0270d11713a697353ae5df1737bc53881d8d1ee7253bcdeac692f64"}
Oct 30 03:53:10.389213 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.389049    4885 scope.go:122] "RemoveContainer" containerID="ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1"
Oct 30 03:53:10.397755 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.397736    4885 scope.go:122] "RemoveContainer" containerID="024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737"
Oct 30 03:53:10.405351 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.405326    4885 scope.go:122] "RemoveContainer" containerID="56a44f5236c5ccd122ccfd9590332e58e14e7e1a3c9ee66bf6f6e89401b74fcc"
Oct 30 03:53:10.412657 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.412636    4885 scope.go:122] "RemoveContainer" containerID="ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1"
Oct 30 03:53:10.412914 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:53:10.412896    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1\": container with ID starting with ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1 not found: ID does not exist" containerID="ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1"
Oct 30 03:53:10.412966 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.412923    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1"} err="failed to get container status \"ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1\": rpc error: code = NotFound desc = could not find container \"ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1\": container with ID starting with ef3819adac9ad8971900febd95e384faa53472ea7c62ab6ab38f691c885d7fd1 not found: ID does not exist"
Oct 30 03:53:10.412966 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.412942    4885 scope.go:122] "RemoveContainer" containerID="024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737"
Oct 30 03:53:10.413169 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:53:10.413150    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737\": container with ID starting with 024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737 not found: ID does not exist" containerID="024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737"
Oct 30 03:53:10.413235 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.413177    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737"} err="failed to get container status \"024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737\": rpc error: code = NotFound desc = could not find container \"024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737\": container with ID starting with 024ec5cb4a9e5d00fe8de0e3a533bdc7949fd3236c792faf84c56850001ad737 not found: ID does not exist"
Oct 30 03:53:10.413235 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.413199    4885 scope.go:122] "RemoveContainer" containerID="56a44f5236c5ccd122ccfd9590332e58e14e7e1a3c9ee66bf6f6e89401b74fcc"
Oct 30 03:53:10.413444 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:53:10.413426    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a44f5236c5ccd122ccfd9590332e58e14e7e1a3c9ee66bf6f6e89401b74fcc\": container with ID starting with 56a44f5236c5ccd122ccfd9590332e58e14e7e1a3c9ee66bf6f6e89401b74fcc not found: ID does not exist" containerID="56a44f5236c5ccd122ccfd9590332e58e14e7e1a3c9ee66bf6f6e89401b74fcc"
Oct 30 03:53:10.413517 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.413453    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a44f5236c5ccd122ccfd9590332e58e14e7e1a3c9ee66bf6f6e89401b74fcc"} err="failed to get container status \"56a44f5236c5ccd122ccfd9590332e58e14e7e1a3c9ee66bf6f6e89401b74fcc\": rpc error: code = NotFound desc = could not find container \"56a44f5236c5ccd122ccfd9590332e58e14e7e1a3c9ee66bf6f6e89401b74fcc\": container with ID starting with 56a44f5236c5ccd122ccfd9590332e58e14e7e1a3c9ee66bf6f6e89401b74fcc not found: ID does not exist"
Oct 30 03:53:10.435100 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.435073    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"]
Oct 30 03:53:10.438928 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.438905    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-predictor-556bb4d79b-x2dmg"]
Oct 30 03:53:10.624244 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:10.624164    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" path="/var/lib/kubelet/pods/33ffed30-6bec-4bdf-a952-3ec9dd3dad2a/volumes"
Oct 30 03:53:25.921921 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:53:25.921883    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:53:30.331279 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:53:30.331252    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:53:55.921762 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:53:55.921718    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:54:25.922654 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:54:25.922597    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:54:34.521531 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.521491    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"]
Oct 30 03:54:34.522363 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.521915    4885 cpu_manager.go:414] "containerMap: removing container" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="storage-initializer"
Oct 30 03:54:34.522363 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.521933    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="storage-initializer"
Oct 30 03:54:34.522363 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.521953    4885 cpu_manager.go:414] "containerMap: removing container" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container"
Oct 30 03:54:34.522363 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.521988    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container"
Oct 30 03:54:34.522363 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.522023    4885 cpu_manager.go:414] "containerMap: removing container" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kube-rbac-proxy"
Oct 30 03:54:34.522363 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.522029    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kube-rbac-proxy"
Oct 30 03:54:34.522363 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.522120    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kserve-container"
Oct 30 03:54:34.522363 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.522129    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ffed30-6bec-4bdf-a952-3ec9dd3dad2a" containerName="kube-rbac-proxy"
Oct 30 03:54:34.525587 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.525567    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.548309 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.548272    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:54:34.548309 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.548297    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:54:34.548309 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.548297    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Oct 30 03:54:34.548734 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.548355    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:54:34.548734 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.548389    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Oct 30 03:54:34.548734 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.548394    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"storage-config\""
Oct 30 03:54:34.551196 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.551160    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"]
Oct 30 03:54:34.640414 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.640377    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c50a6023-a652-4aed-a802-1a34d2563a60-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.640414 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.640417    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c50a6023-a652-4aed-a802-1a34d2563a60-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.640660 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.640442    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nnnt\" (UniqueName: \"kubernetes.io/projected/c50a6023-a652-4aed-a802-1a34d2563a60-kube-api-access-7nnnt\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.640660 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.640542    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c50a6023-a652-4aed-a802-1a34d2563a60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.741143 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.741107    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c50a6023-a652-4aed-a802-1a34d2563a60-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.741347 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.741150    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c50a6023-a652-4aed-a802-1a34d2563a60-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.741347 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.741190    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nnnt\" (UniqueName: \"kubernetes.io/projected/c50a6023-a652-4aed-a802-1a34d2563a60-kube-api-access-7nnnt\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.741347 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.741227    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c50a6023-a652-4aed-a802-1a34d2563a60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.741606 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.741577    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c50a6023-a652-4aed-a802-1a34d2563a60-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.742012 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.741988    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c50a6023-a652-4aed-a802-1a34d2563a60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.744263 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.744240    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c50a6023-a652-4aed-a802-1a34d2563a60-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.763255 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.763219    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nnnt\" (UniqueName: \"kubernetes.io/projected/c50a6023-a652-4aed-a802-1a34d2563a60-kube-api-access-7nnnt\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:34.842597 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:34.842523    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:35.146908 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:35.146814    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"]
Oct 30 03:54:35.150471 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:54:35.150441    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50a6023_a652_4aed_a802_1a34d2563a60.slice/crio-6f9d500590d2c877923c3c6d5204271ff135269add1ea81e130251cfa74c1f20 WatchSource:0}: Error finding container 6f9d500590d2c877923c3c6d5204271ff135269add1ea81e130251cfa74c1f20: Status 404 returned error can't find the container with id 6f9d500590d2c877923c3c6d5204271ff135269add1ea81e130251cfa74c1f20
Oct 30 03:54:35.753997 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:35.753959    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm" event={"ID":"c50a6023-a652-4aed-a802-1a34d2563a60","Type":"ContainerStarted","Data":"4689be6038d60c9368e0d1ec73106cf990dc096e4e3c74180375c81dd648bb68"}
Oct 30 03:54:35.754465 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:35.754031    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm" event={"ID":"c50a6023-a652-4aed-a802-1a34d2563a60","Type":"ContainerStarted","Data":"6f9d500590d2c877923c3c6d5204271ff135269add1ea81e130251cfa74c1f20"}
Oct 30 03:54:38.768900 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:38.768875    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm_c50a6023-a652-4aed-a802-1a34d2563a60/storage-initializer/0.log"
Oct 30 03:54:38.769285 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:38.768911    4885 generic.go:356] "Generic (PLEG): container finished" podID="c50a6023-a652-4aed-a802-1a34d2563a60" containerID="4689be6038d60c9368e0d1ec73106cf990dc096e4e3c74180375c81dd648bb68" exitCode=1
Oct 30 03:54:38.769285 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:38.768985    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm" event={"ID":"c50a6023-a652-4aed-a802-1a34d2563a60","Type":"ContainerDied","Data":"4689be6038d60c9368e0d1ec73106cf990dc096e4e3c74180375c81dd648bb68"}
Oct 30 03:54:39.774699 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:39.774672    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm_c50a6023-a652-4aed-a802-1a34d2563a60/storage-initializer/0.log"
Oct 30 03:54:39.775116 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:39.774762    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm" event={"ID":"c50a6023-a652-4aed-a802-1a34d2563a60","Type":"ContainerStarted","Data":"6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd"}
Oct 30 03:54:44.485935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.485904    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"]
Oct 30 03:54:44.486417 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.486112    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm" podUID="c50a6023-a652-4aed-a802-1a34d2563a60" containerName="storage-initializer" containerID="cri-o://6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd" gracePeriod=30
Oct 30 03:54:44.621933 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.621908    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm_c50a6023-a652-4aed-a802-1a34d2563a60/storage-initializer/1.log"
Oct 30 03:54:44.622338 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.622320    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm_c50a6023-a652-4aed-a802-1a34d2563a60/storage-initializer/0.log"
Oct 30 03:54:44.624717 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.624700    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:44.726368 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.726336    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50a6023-a652-4aed-a802-1a34d2563a60-kube-api-access-7nnnt\" (UniqueName: \"kubernetes.io/projected/c50a6023-a652-4aed-a802-1a34d2563a60-kube-api-access-7nnnt\") pod \"c50a6023-a652-4aed-a802-1a34d2563a60\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") "
Oct 30 03:54:44.726581 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.726385    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50a6023-a652-4aed-a802-1a34d2563a60-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c50a6023-a652-4aed-a802-1a34d2563a60-kube-rbac-proxy-sar-config\") pod \"c50a6023-a652-4aed-a802-1a34d2563a60\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") "
Oct 30 03:54:44.726581 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.726441    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50a6023-a652-4aed-a802-1a34d2563a60-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c50a6023-a652-4aed-a802-1a34d2563a60-kserve-provision-location\") pod \"c50a6023-a652-4aed-a802-1a34d2563a60\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") "
Oct 30 03:54:44.726581 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.726471    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50a6023-a652-4aed-a802-1a34d2563a60-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c50a6023-a652-4aed-a802-1a34d2563a60-proxy-tls\") pod \"c50a6023-a652-4aed-a802-1a34d2563a60\" (UID: \"c50a6023-a652-4aed-a802-1a34d2563a60\") "
Oct 30 03:54:44.726792 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.726763    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50a6023-a652-4aed-a802-1a34d2563a60-kserve-provision-location" pod "c50a6023-a652-4aed-a802-1a34d2563a60" (UID: "c50a6023-a652-4aed-a802-1a34d2563a60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:54:44.726900 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.726870    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50a6023-a652-4aed-a802-1a34d2563a60-kube-rbac-proxy-sar-config" pod "c50a6023-a652-4aed-a802-1a34d2563a60" (UID: "c50a6023-a652-4aed-a802-1a34d2563a60"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:54:44.728783 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.728761    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50a6023-a652-4aed-a802-1a34d2563a60-proxy-tls" pod "c50a6023-a652-4aed-a802-1a34d2563a60" (UID: "c50a6023-a652-4aed-a802-1a34d2563a60"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:54:44.728783 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.728764    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50a6023-a652-4aed-a802-1a34d2563a60-kube-api-access-7nnnt" pod "c50a6023-a652-4aed-a802-1a34d2563a60" (UID: "c50a6023-a652-4aed-a802-1a34d2563a60"). InnerVolumeSpecName "kube-api-access-7nnnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:54:44.796480 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.796410    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm_c50a6023-a652-4aed-a802-1a34d2563a60/storage-initializer/1.log"
Oct 30 03:54:44.796841 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.796827    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm_c50a6023-a652-4aed-a802-1a34d2563a60/storage-initializer/0.log"
Oct 30 03:54:44.796914 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.796861    4885 generic.go:356] "Generic (PLEG): container finished" podID="c50a6023-a652-4aed-a802-1a34d2563a60" containerID="6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd" exitCode=1
Oct 30 03:54:44.796914 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.796892    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm" event={"ID":"c50a6023-a652-4aed-a802-1a34d2563a60","Type":"ContainerDied","Data":"6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd"}
Oct 30 03:54:44.797035 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.796943    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"
Oct 30 03:54:44.797035 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.796962    4885 scope.go:122] "RemoveContainer" containerID="6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd"
Oct 30 03:54:44.797136 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.796952    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm" event={"ID":"c50a6023-a652-4aed-a802-1a34d2563a60","Type":"ContainerDied","Data":"6f9d500590d2c877923c3c6d5204271ff135269add1ea81e130251cfa74c1f20"}
Oct 30 03:54:44.806006 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.805990    4885 scope.go:122] "RemoveContainer" containerID="4689be6038d60c9368e0d1ec73106cf990dc096e4e3c74180375c81dd648bb68"
Oct 30 03:54:44.813473 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.813457    4885 scope.go:122] "RemoveContainer" containerID="6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd"
Oct 30 03:54:44.813744 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:54:44.813723    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd\": container with ID starting with 6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd not found: ID does not exist" containerID="6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd"
Oct 30 03:54:44.813807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.813753    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd"} err="failed to get container status \"6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd\": rpc error: code = NotFound desc = could not find container \"6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd\": container with ID starting with 6343b7984681f4beafabdf46c6d2028695d0c1f8d250d4130646541e63b4e0cd not found: ID does not exist"
Oct 30 03:54:44.813807 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.813772    4885 scope.go:122] "RemoveContainer" containerID="4689be6038d60c9368e0d1ec73106cf990dc096e4e3c74180375c81dd648bb68"
Oct 30 03:54:44.813996 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:54:44.813978    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4689be6038d60c9368e0d1ec73106cf990dc096e4e3c74180375c81dd648bb68\": container with ID starting with 4689be6038d60c9368e0d1ec73106cf990dc096e4e3c74180375c81dd648bb68 not found: ID does not exist" containerID="4689be6038d60c9368e0d1ec73106cf990dc096e4e3c74180375c81dd648bb68"
Oct 30 03:54:44.814050 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.814007    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4689be6038d60c9368e0d1ec73106cf990dc096e4e3c74180375c81dd648bb68"} err="failed to get container status \"4689be6038d60c9368e0d1ec73106cf990dc096e4e3c74180375c81dd648bb68\": rpc error: code = NotFound desc = could not find container \"4689be6038d60c9368e0d1ec73106cf990dc096e4e3c74180375c81dd648bb68\": container with ID starting with 4689be6038d60c9368e0d1ec73106cf990dc096e4e3c74180375c81dd648bb68 not found: ID does not exist"
Oct 30 03:54:44.827707 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.827688    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nnnt\" (UniqueName: \"kubernetes.io/projected/c50a6023-a652-4aed-a802-1a34d2563a60-kube-api-access-7nnnt\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:54:44.827793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.827710    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c50a6023-a652-4aed-a802-1a34d2563a60-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:54:44.827793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.827724    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c50a6023-a652-4aed-a802-1a34d2563a60-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:54:44.827793 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.827738    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c50a6023-a652-4aed-a802-1a34d2563a60-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:54:44.868143 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.868120    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"]
Oct 30 03:54:44.871906 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:44.871885    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-fail-predictor-754d5f6b-vzhnm"]
Oct 30 03:54:46.624547 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:46.624518    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50a6023-a652-4aed-a802-1a34d2563a60" path="/var/lib/kubelet/pods/c50a6023-a652-4aed-a802-1a34d2563a60/volumes"
Oct 30 03:54:54.978996 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:54.978968    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:54:54.984338 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:54.984317    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:54:54.985526 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:54.985504    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:54:54.990661 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:54:54.990643    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:54:55.921125 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:54:55.921091    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:55:25.920695 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:55:25.920660    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:55:48.661899 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:48.661863    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"]
Oct 30 03:55:48.662815 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:48.662170    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2" podUID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerName="kserve-container" containerID="cri-o://e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb" gracePeriod=30
Oct 30 03:55:48.662815 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:48.662239    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2" podUID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerName="kube-rbac-proxy" containerID="cri-o://796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350" gracePeriod=30
Oct 30 03:55:49.070312 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.070280    4885 generic.go:356] "Generic (PLEG): container finished" podID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerID="796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350" exitCode=0
Oct 30 03:55:49.070509 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.070362    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2" event={"ID":"cee2545e-7520-4c0c-8c7b-123738f9177c","Type":"ContainerDied","Data":"796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350"}
Oct 30 03:55:49.716365 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.716342    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:55:49.851938 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.851827    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-api-access-t7j8t\" (UniqueName: \"kubernetes.io/projected/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-api-access-t7j8t\") pod \"cee2545e-7520-4c0c-8c7b-123738f9177c\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") "
Oct 30 03:55:49.851938 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.851912    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee2545e-7520-4c0c-8c7b-123738f9177c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cee2545e-7520-4c0c-8c7b-123738f9177c-kserve-provision-location\") pod \"cee2545e-7520-4c0c-8c7b-123738f9177c\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") "
Oct 30 03:55:49.852177 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.851969    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-cabundle-cert\") pod \"cee2545e-7520-4c0c-8c7b-123738f9177c\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") "
Oct 30 03:55:49.852177 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.852008    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-rbac-proxy-sar-config\") pod \"cee2545e-7520-4c0c-8c7b-123738f9177c\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") "
Oct 30 03:55:49.852177 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.852049    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee2545e-7520-4c0c-8c7b-123738f9177c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/cee2545e-7520-4c0c-8c7b-123738f9177c-proxy-tls\") pod \"cee2545e-7520-4c0c-8c7b-123738f9177c\" (UID: \"cee2545e-7520-4c0c-8c7b-123738f9177c\") "
Oct 30 03:55:49.852340 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.852311    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee2545e-7520-4c0c-8c7b-123738f9177c-kserve-provision-location" pod "cee2545e-7520-4c0c-8c7b-123738f9177c" (UID: "cee2545e-7520-4c0c-8c7b-123738f9177c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:55:49.852393 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.852350    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-cabundle-cert" pod "cee2545e-7520-4c0c-8c7b-123738f9177c" (UID: "cee2545e-7520-4c0c-8c7b-123738f9177c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:55:49.852393 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.852372    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-rbac-proxy-sar-config" pod "cee2545e-7520-4c0c-8c7b-123738f9177c" (UID: "cee2545e-7520-4c0c-8c7b-123738f9177c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:55:49.854298 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.854270    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-api-access-t7j8t" pod "cee2545e-7520-4c0c-8c7b-123738f9177c" (UID: "cee2545e-7520-4c0c-8c7b-123738f9177c"). InnerVolumeSpecName "kube-api-access-t7j8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:55:49.854446 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.854329    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee2545e-7520-4c0c-8c7b-123738f9177c-proxy-tls" pod "cee2545e-7520-4c0c-8c7b-123738f9177c" (UID: "cee2545e-7520-4c0c-8c7b-123738f9177c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:55:49.953298 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.953262    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cee2545e-7520-4c0c-8c7b-123738f9177c-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:55:49.953298 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.953292    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:55:49.953298 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.953302    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:55:49.953522 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.953311    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cee2545e-7520-4c0c-8c7b-123738f9177c-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:55:49.953522 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:49.953321    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7j8t\" (UniqueName: \"kubernetes.io/projected/cee2545e-7520-4c0c-8c7b-123738f9177c-kube-api-access-t7j8t\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:55:50.077233 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.077201    4885 generic.go:356] "Generic (PLEG): container finished" podID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerID="e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb" exitCode=0
Oct 30 03:55:50.077421 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.077263    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"
Oct 30 03:55:50.077421 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.077277    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2" event={"ID":"cee2545e-7520-4c0c-8c7b-123738f9177c","Type":"ContainerDied","Data":"e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb"}
Oct 30 03:55:50.077421 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.077326    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2" event={"ID":"cee2545e-7520-4c0c-8c7b-123738f9177c","Type":"ContainerDied","Data":"2a5d52b475591bc16b48c4b24cf9a6fe74b2b24480d6da784caba431cc634b42"}
Oct 30 03:55:50.077421 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.077351    4885 scope.go:122] "RemoveContainer" containerID="796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350"
Oct 30 03:55:50.087001 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.086981    4885 scope.go:122] "RemoveContainer" containerID="e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb"
Oct 30 03:55:50.095079 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.095061    4885 scope.go:122] "RemoveContainer" containerID="b5bc9be19757d496e9fb2b7b15337e3ef4535f0cf0f89fffaae6e0dcd0140166"
Oct 30 03:55:50.103123 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.103106    4885 scope.go:122] "RemoveContainer" containerID="796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350"
Oct 30 03:55:50.103496 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:55:50.103470    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350\": container with ID starting with 796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350 not found: ID does not exist" containerID="796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350"
Oct 30 03:55:50.103570 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.103506    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350"} err="failed to get container status \"796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350\": rpc error: code = NotFound desc = could not find container \"796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350\": container with ID starting with 796b9720f463700d75276647ca5dd796e542bd9e96e9d9ab4ab93668d9398350 not found: ID does not exist"
Oct 30 03:55:50.103570 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.103525    4885 scope.go:122] "RemoveContainer" containerID="e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb"
Oct 30 03:55:50.103810 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:55:50.103792    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb\": container with ID starting with e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb not found: ID does not exist" containerID="e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb"
Oct 30 03:55:50.103872 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.103819    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb"} err="failed to get container status \"e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb\": rpc error: code = NotFound desc = could not find container \"e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb\": container with ID starting with e8fc4b7395688b7659b4fd4198de02668e17e25faa1ea74033f43a6b719ccfbb not found: ID does not exist"
Oct 30 03:55:50.103872 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.103846    4885 scope.go:122] "RemoveContainer" containerID="b5bc9be19757d496e9fb2b7b15337e3ef4535f0cf0f89fffaae6e0dcd0140166"
Oct 30 03:55:50.104077 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:55:50.104059    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5bc9be19757d496e9fb2b7b15337e3ef4535f0cf0f89fffaae6e0dcd0140166\": container with ID starting with b5bc9be19757d496e9fb2b7b15337e3ef4535f0cf0f89fffaae6e0dcd0140166 not found: ID does not exist" containerID="b5bc9be19757d496e9fb2b7b15337e3ef4535f0cf0f89fffaae6e0dcd0140166"
Oct 30 03:55:50.104128 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.104081    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5bc9be19757d496e9fb2b7b15337e3ef4535f0cf0f89fffaae6e0dcd0140166"} err="failed to get container status \"b5bc9be19757d496e9fb2b7b15337e3ef4535f0cf0f89fffaae6e0dcd0140166\": rpc error: code = NotFound desc = could not find container \"b5bc9be19757d496e9fb2b7b15337e3ef4535f0cf0f89fffaae6e0dcd0140166\": container with ID starting with b5bc9be19757d496e9fb2b7b15337e3ef4535f0cf0f89fffaae6e0dcd0140166 not found: ID does not exist"
Oct 30 03:55:50.123357 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.123330    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"]
Oct 30 03:55:50.133808 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.133776    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-v2-runtime-predictor-75975dbcd5-vrwc2"]
Oct 30 03:55:50.624281 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:50.624251    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee2545e-7520-4c0c-8c7b-123738f9177c" path="/var/lib/kubelet/pods/cee2545e-7520-4c0c-8c7b-123738f9177c/volumes"
Oct 30 03:55:52.198391 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198359    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"]
Oct 30 03:55:52.198818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198696    4885 cpu_manager.go:414] "containerMap: removing container" podUID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerName="storage-initializer"
Oct 30 03:55:52.198818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198708    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerName="storage-initializer"
Oct 30 03:55:52.198818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198722    4885 cpu_manager.go:414] "containerMap: removing container" podUID="c50a6023-a652-4aed-a802-1a34d2563a60" containerName="storage-initializer"
Oct 30 03:55:52.198818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198731    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50a6023-a652-4aed-a802-1a34d2563a60" containerName="storage-initializer"
Oct 30 03:55:52.198818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198744    4885 cpu_manager.go:414] "containerMap: removing container" podUID="c50a6023-a652-4aed-a802-1a34d2563a60" containerName="storage-initializer"
Oct 30 03:55:52.198818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198751    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50a6023-a652-4aed-a802-1a34d2563a60" containerName="storage-initializer"
Oct 30 03:55:52.198818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198767    4885 cpu_manager.go:414] "containerMap: removing container" podUID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerName="kube-rbac-proxy"
Oct 30 03:55:52.198818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198775    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerName="kube-rbac-proxy"
Oct 30 03:55:52.198818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198799    4885 cpu_manager.go:414] "containerMap: removing container" podUID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerName="kserve-container"
Oct 30 03:55:52.198818 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198806    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerName="kserve-container"
Oct 30 03:55:52.199285 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198914    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerName="kserve-container"
Oct 30 03:55:52.199285 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198934    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50a6023-a652-4aed-a802-1a34d2563a60" containerName="storage-initializer"
Oct 30 03:55:52.199285 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198943    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50a6023-a652-4aed-a802-1a34d2563a60" containerName="storage-initializer"
Oct 30 03:55:52.199285 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.198953    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee2545e-7520-4c0c-8c7b-123738f9177c" containerName="kube-rbac-proxy"
Oct 30 03:55:52.203788 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.203769    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.219381 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.219362    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:55:52.229398 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.229378    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:55:52.229547 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.229382    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:55:52.229547 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.229389    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:55:52.229747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.229399    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Oct 30 03:55:52.229747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.229440    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Oct 30 03:55:52.229747 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.229503    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:55:52.230995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.230975    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"]
Oct 30 03:55:52.273168 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.273147    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtvc2\" (UniqueName: \"kubernetes.io/projected/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-api-access-mtvc2\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.273274 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.273176    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.273326 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.273284    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.273365 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.273336    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db95b5f6-8ea6-4e26-8115-f9703b826ca9-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.273400 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.273368    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.374431 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.374405    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db95b5f6-8ea6-4e26-8115-f9703b826ca9-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.374528 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.374436    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.374528 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.374461    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtvc2\" (UniqueName: \"kubernetes.io/projected/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-api-access-mtvc2\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.374528 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.374480    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.374710 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.374535    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.374987 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.374967    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.375204 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.375165    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.375204 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.375165    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.377016 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.376995    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db95b5f6-8ea6-4e26-8115-f9703b826ca9-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.398300 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.398277    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtvc2\" (UniqueName: \"kubernetes.io/projected/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-api-access-mtvc2\") pod \"isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.520805 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.520775    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:55:52.825512 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:52.825395    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"]
Oct 30 03:55:52.831842 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:55:52.831801    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb95b5f6_8ea6_4e26_8115_f9703b826ca9.slice/crio-93a0988a70979fe4eff63511bdb0f6df0d738f58ca822ba30052947071e2c058 WatchSource:0}: Error finding container 93a0988a70979fe4eff63511bdb0f6df0d738f58ca822ba30052947071e2c058: Status 404 returned error can't find the container with id 93a0988a70979fe4eff63511bdb0f6df0d738f58ca822ba30052947071e2c058
Oct 30 03:55:53.092197 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:53.092116    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" event={"ID":"db95b5f6-8ea6-4e26-8115-f9703b826ca9","Type":"ContainerStarted","Data":"932342330f5821853012c451470bdb9b8fe9ad85cfd8f7ba63c289ff568e4148"}
Oct 30 03:55:53.092197 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:53.092170    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" event={"ID":"db95b5f6-8ea6-4e26-8115-f9703b826ca9","Type":"ContainerStarted","Data":"93a0988a70979fe4eff63511bdb0f6df0d738f58ca822ba30052947071e2c058"}
Oct 30 03:55:55.921870 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:55:55.921833    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:55:57.109803 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:57.109774    4885 generic.go:356] "Generic (PLEG): container finished" podID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerID="932342330f5821853012c451470bdb9b8fe9ad85cfd8f7ba63c289ff568e4148" exitCode=0
Oct 30 03:55:57.110210 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:57.109852    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" event={"ID":"db95b5f6-8ea6-4e26-8115-f9703b826ca9","Type":"ContainerDied","Data":"932342330f5821853012c451470bdb9b8fe9ad85cfd8f7ba63c289ff568e4148"}
Oct 30 03:55:57.111119 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:55:57.111105    4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:56:04.149412 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:04.149378    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" event={"ID":"db95b5f6-8ea6-4e26-8115-f9703b826ca9","Type":"ContainerStarted","Data":"6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461"}
Oct 30 03:56:04.149871 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:04.149446    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" event={"ID":"db95b5f6-8ea6-4e26-8115-f9703b826ca9","Type":"ContainerStarted","Data":"2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af"}
Oct 30 03:56:04.149871 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:04.149640    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:56:04.201037 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:04.200992    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" podStartSLOduration=2.155037113 podStartE2EDuration="12.20097586s" totalImagesPullingTime="6.369488517s" totalInitContainerRuntime="3.67645023s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:04.200343764 +0000 UTC m=+3669.710512808" watchObservedRunningTime="2026-10-30 03:56:04.20097586 +0000 UTC m=+3669.711144886"
Oct 30 03:56:05.154588 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:05.154557    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:56:05.155876 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:05.155849    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.90:8080: connect: connection refused"
Oct 30 03:56:06.159084 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:06.159037    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.90:8080: connect: connection refused"
Oct 30 03:56:11.163437 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:11.163411    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:56:11.163944 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:11.163921    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.90:8080: connect: connection refused"
Oct 30 03:56:17.452977 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.452939    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"]
Oct 30 03:56:17.456888 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.456866    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.484456 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.484432    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Oct 30 03:56:17.487744 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.487727    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"storage-config\""
Oct 30 03:56:17.487744 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.487736    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Oct 30 03:56:17.487912 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.487757    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:56:17.487912 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.487773    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:56:17.487912 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.487763    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:56:17.490742 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.490722    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"]
Oct 30 03:56:17.508417 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.508386    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.508542 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.508421    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.508542 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.508449    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.508653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.508630    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln546\" (UniqueName: \"kubernetes.io/projected/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-api-access-ln546\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.609246 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.609209    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln546\" (UniqueName: \"kubernetes.io/projected/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-api-access-ln546\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.609419 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.609270    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.609419 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.609290    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.609419 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.609328    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.609717 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.609696    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.610091 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.610068    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.611799 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.611776    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.631297 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.631269    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln546\" (UniqueName: \"kubernetes.io/projected/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-api-access-ln546\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:17.773787 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:17.773758    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:18.080173 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:18.080083    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"]
Oct 30 03:56:18.083311 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:56:18.083286    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173401a9_067f_4b30_b2e4_f9c28ac3dc8b.slice/crio-4709e58508af664556a785aee7ceb82ff07c3f14d35994ea658a62fe53b398a5 WatchSource:0}: Error finding container 4709e58508af664556a785aee7ceb82ff07c3f14d35994ea658a62fe53b398a5: Status 404 returned error can't find the container with id 4709e58508af664556a785aee7ceb82ff07c3f14d35994ea658a62fe53b398a5
Oct 30 03:56:18.213277 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:18.213235    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn" event={"ID":"173401a9-067f-4b30-b2e4-f9c28ac3dc8b","Type":"ContainerStarted","Data":"27cc691f931c09c11a8678f86b282824cf9212d3694975881c7d60c379f7ea70"}
Oct 30 03:56:18.213455 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:18.213306    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn" event={"ID":"173401a9-067f-4b30-b2e4-f9c28ac3dc8b","Type":"ContainerStarted","Data":"4709e58508af664556a785aee7ceb82ff07c3f14d35994ea658a62fe53b398a5"}
Oct 30 03:56:21.164718 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:21.164690    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:56:25.246249 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:25.246218    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn_173401a9-067f-4b30-b2e4-f9c28ac3dc8b/storage-initializer/0.log"
Oct 30 03:56:25.246693 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:25.246257    4885 generic.go:356] "Generic (PLEG): container finished" podID="173401a9-067f-4b30-b2e4-f9c28ac3dc8b" containerID="27cc691f931c09c11a8678f86b282824cf9212d3694975881c7d60c379f7ea70" exitCode=1
Oct 30 03:56:25.246693 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:25.246316    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn" event={"ID":"173401a9-067f-4b30-b2e4-f9c28ac3dc8b","Type":"ContainerDied","Data":"27cc691f931c09c11a8678f86b282824cf9212d3694975881c7d60c379f7ea70"}
Oct 30 03:56:25.923191 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:56:25.923153    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:56:26.252906 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:26.252880    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn_173401a9-067f-4b30-b2e4-f9c28ac3dc8b/storage-initializer/0.log"
Oct 30 03:56:26.253305 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:26.252961    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn" event={"ID":"173401a9-067f-4b30-b2e4-f9c28ac3dc8b","Type":"ContainerStarted","Data":"4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf"}
Oct 30 03:56:26.573794 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:26.573715    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"]
Oct 30 03:56:26.573995 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:26.573970    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="kserve-container" containerID="cri-o://2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af" gracePeriod=30
Oct 30 03:56:26.574054 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:26.573987    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="kube-rbac-proxy" containerID="cri-o://6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461" gracePeriod=30
Oct 30 03:56:27.259600 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.259571    4885 generic.go:356] "Generic (PLEG): container finished" podID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerID="6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461" exitCode=0
Oct 30 03:56:27.260006 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.259606    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" event={"ID":"db95b5f6-8ea6-4e26-8115-f9703b826ca9","Type":"ContainerDied","Data":"6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461"}
Oct 30 03:56:27.416499 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.416454    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"]
Oct 30 03:56:27.416775 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.416749    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn" podUID="173401a9-067f-4b30-b2e4-f9c28ac3dc8b" containerName="storage-initializer" containerID="cri-o://4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf" gracePeriod=30
Oct 30 03:56:27.438363 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.438338    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:56:27.495391 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.495298    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-api-access-mtvc2\" (UniqueName: \"kubernetes.io/projected/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-api-access-mtvc2\") pod \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") "
Oct 30 03:56:27.495391 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.495375    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-rbac-proxy-sar-config\") pod \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") "
Oct 30 03:56:27.495661 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.495473    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db95b5f6-8ea6-4e26-8115-f9703b826ca9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/db95b5f6-8ea6-4e26-8115-f9703b826ca9-proxy-tls\") pod \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") "
Oct 30 03:56:27.495661 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.495558    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kserve-provision-location\") pod \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") "
Oct 30 03:56:27.495661 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.495627    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-cabundle-cert\") pod \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\" (UID: \"db95b5f6-8ea6-4e26-8115-f9703b826ca9\") "
Oct 30 03:56:27.496897 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.495822    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-rbac-proxy-sar-config" pod "db95b5f6-8ea6-4e26-8115-f9703b826ca9" (UID: "db95b5f6-8ea6-4e26-8115-f9703b826ca9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:27.496897 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.496058    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kserve-provision-location" pod "db95b5f6-8ea6-4e26-8115-f9703b826ca9" (UID: "db95b5f6-8ea6-4e26-8115-f9703b826ca9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:27.496897 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.496197    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-cabundle-cert" pod "db95b5f6-8ea6-4e26-8115-f9703b826ca9" (UID: "db95b5f6-8ea6-4e26-8115-f9703b826ca9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:27.498130 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.498085    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-api-access-mtvc2" pod "db95b5f6-8ea6-4e26-8115-f9703b826ca9" (UID: "db95b5f6-8ea6-4e26-8115-f9703b826ca9"). InnerVolumeSpecName "kube-api-access-mtvc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:27.498260 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.498185    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db95b5f6-8ea6-4e26-8115-f9703b826ca9-proxy-tls" pod "db95b5f6-8ea6-4e26-8115-f9703b826ca9" (UID: "db95b5f6-8ea6-4e26-8115-f9703b826ca9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:27.596554 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.596521    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:56:27.596554 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.596552    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:56:27.596554 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.596562    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtvc2\" (UniqueName: \"kubernetes.io/projected/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-api-access-mtvc2\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:56:27.596790 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.596571    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db95b5f6-8ea6-4e26-8115-f9703b826ca9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:56:27.596790 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:27.596580    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db95b5f6-8ea6-4e26-8115-f9703b826ca9-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:56:28.265800 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.265760    4885 generic.go:356] "Generic (PLEG): container finished" podID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerID="2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af" exitCode=0
Oct 30 03:56:28.266222 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.265841    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" event={"ID":"db95b5f6-8ea6-4e26-8115-f9703b826ca9","Type":"ContainerDied","Data":"2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af"}
Oct 30 03:56:28.266222 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.265855    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"
Oct 30 03:56:28.266222 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.265927    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl" event={"ID":"db95b5f6-8ea6-4e26-8115-f9703b826ca9","Type":"ContainerDied","Data":"93a0988a70979fe4eff63511bdb0f6df0d738f58ca822ba30052947071e2c058"}
Oct 30 03:56:28.266222 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.265954    4885 scope.go:122] "RemoveContainer" containerID="6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461"
Oct 30 03:56:28.275198 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.275179    4885 scope.go:122] "RemoveContainer" containerID="2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af"
Oct 30 03:56:28.283597 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.283575    4885 scope.go:122] "RemoveContainer" containerID="932342330f5821853012c451470bdb9b8fe9ad85cfd8f7ba63c289ff568e4148"
Oct 30 03:56:28.291700 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.291673    4885 scope.go:122] "RemoveContainer" containerID="6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461"
Oct 30 03:56:28.292034 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:56:28.292008    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461\": container with ID starting with 6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461 not found: ID does not exist" containerID="6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461"
Oct 30 03:56:28.292124 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.292043    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461"} err="failed to get container status \"6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461\": rpc error: code = NotFound desc = could not find container \"6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461\": container with ID starting with 6655d3b09e2e0cda85dc9ce97c14cb0c850824c6dcd6bc6417422539b734d461 not found: ID does not exist"
Oct 30 03:56:28.292124 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.292062    4885 scope.go:122] "RemoveContainer" containerID="2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af"
Oct 30 03:56:28.292301 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:56:28.292283    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af\": container with ID starting with 2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af not found: ID does not exist" containerID="2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af"
Oct 30 03:56:28.292358 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.292305    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af"} err="failed to get container status \"2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af\": rpc error: code = NotFound desc = could not find container \"2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af\": container with ID starting with 2d8d025719c2d5c92340313c73a3e80c9420bdd4945e4b9d87568e12d3e736af not found: ID does not exist"
Oct 30 03:56:28.292358 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.292318    4885 scope.go:122] "RemoveContainer" containerID="932342330f5821853012c451470bdb9b8fe9ad85cfd8f7ba63c289ff568e4148"
Oct 30 03:56:28.292522 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:56:28.292504    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932342330f5821853012c451470bdb9b8fe9ad85cfd8f7ba63c289ff568e4148\": container with ID starting with 932342330f5821853012c451470bdb9b8fe9ad85cfd8f7ba63c289ff568e4148 not found: ID does not exist" containerID="932342330f5821853012c451470bdb9b8fe9ad85cfd8f7ba63c289ff568e4148"
Oct 30 03:56:28.292581 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.292528    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932342330f5821853012c451470bdb9b8fe9ad85cfd8f7ba63c289ff568e4148"} err="failed to get container status \"932342330f5821853012c451470bdb9b8fe9ad85cfd8f7ba63c289ff568e4148\": rpc error: code = NotFound desc = could not find container \"932342330f5821853012c451470bdb9b8fe9ad85cfd8f7ba63c289ff568e4148\": container with ID starting with 932342330f5821853012c451470bdb9b8fe9ad85cfd8f7ba63c289ff568e4148 not found: ID does not exist"
Oct 30 03:56:28.311654 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.311605    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"]
Oct 30 03:56:28.321702 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.321666    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-v2-kserve-predictor-84fcdc566d-whdjl"]
Oct 30 03:56:28.623553 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:28.623476    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" path="/var/lib/kubelet/pods/db95b5f6-8ea6-4e26-8115-f9703b826ca9/volumes"
Oct 30 03:56:30.075773 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.075718    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"]
Oct 30 03:56:30.076248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.076136    4885 cpu_manager.go:414] "containerMap: removing container" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="kube-rbac-proxy"
Oct 30 03:56:30.076248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.076151    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="kube-rbac-proxy"
Oct 30 03:56:30.076248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.076171    4885 cpu_manager.go:414] "containerMap: removing container" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="storage-initializer"
Oct 30 03:56:30.076248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.076177    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="storage-initializer"
Oct 30 03:56:30.076248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.076188    4885 cpu_manager.go:414] "containerMap: removing container" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="kserve-container"
Oct 30 03:56:30.076248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.076194    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="kserve-container"
Oct 30 03:56:30.076248 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.076251    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="kserve-container"
Oct 30 03:56:30.076465 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.076263    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="db95b5f6-8ea6-4e26-8115-f9703b826ca9" containerName="kube-rbac-proxy"
Oct 30 03:56:30.081225 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.081206    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.106366 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.106224    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:56:30.106366 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.106259    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:56:30.106366 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.106280    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:56:30.106366 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.106305    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Oct 30 03:56:30.106366 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.106315    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:56:30.106366 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.106314    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Oct 30 03:56:30.106366 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.106320    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:56:30.107871 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.107851    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"]
Oct 30 03:56:30.218168 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.218132    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.218168 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.218170    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msbvm\" (UniqueName: \"kubernetes.io/projected/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-api-access-msbvm\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.218442 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.218202    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.218442 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.218309    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.218442 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.218377    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.319087 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.319048    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.319087 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.319085    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msbvm\" (UniqueName: \"kubernetes.io/projected/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-api-access-msbvm\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.319345 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.319103    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.319345 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.319225    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.319345 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.319301    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.319605 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.319583    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.319780 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.319756    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.319890 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.319863    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.322103 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.322080    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.340808 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.340735    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msbvm\" (UniqueName: \"kubernetes.io/projected/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-api-access-msbvm\") pod \"isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") " pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.399526 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.399486    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:30.710592 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:30.710440    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"]
Oct 30 03:56:30.713418 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:56:30.713383    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b243f7_4c85_4198_b1ac_1aaae8bab7e7.slice/crio-53c5f4f28c0079a203a6c7af1de616727779134817f123c5c7a34b3044e3568e WatchSource:0}: Error finding container 53c5f4f28c0079a203a6c7af1de616727779134817f123c5c7a34b3044e3568e: Status 404 returned error can't find the container with id 53c5f4f28c0079a203a6c7af1de616727779134817f123c5c7a34b3044e3568e
Oct 30 03:56:31.280940 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:31.280909    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz" event={"ID":"09b243f7-4c85-4198-b1ac-1aaae8bab7e7","Type":"ContainerStarted","Data":"08020f4c35d79aceb97e24c0d9bb049fa987bc6903dff520a2a5a1d1ccd8d3fb"}
Oct 30 03:56:31.281422 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:31.280975    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz" event={"ID":"09b243f7-4c85-4198-b1ac-1aaae8bab7e7","Type":"ContainerStarted","Data":"53c5f4f28c0079a203a6c7af1de616727779134817f123c5c7a34b3044e3568e"}
Oct 30 03:56:32.866476 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:32.866451    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn_173401a9-067f-4b30-b2e4-f9c28ac3dc8b/storage-initializer/1.log"
Oct 30 03:56:32.866918 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:32.866880    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn_173401a9-067f-4b30-b2e4-f9c28ac3dc8b/storage-initializer/0.log"
Oct 30 03:56:32.869087 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:32.869069    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:32.942178 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:32.942093    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-proxy-tls\") pod \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") "
Oct 30 03:56:32.942178 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:32.942138    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-rbac-proxy-sar-config\") pod \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") "
Oct 30 03:56:32.942424 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:32.942185    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-api-access-ln546\" (UniqueName: \"kubernetes.io/projected/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-api-access-ln546\") pod \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") "
Oct 30 03:56:32.942424 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:32.942243    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kserve-provision-location\") pod \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\" (UID: \"173401a9-067f-4b30-b2e4-f9c28ac3dc8b\") "
Oct 30 03:56:32.942564 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:32.942538    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kserve-provision-location" pod "173401a9-067f-4b30-b2e4-f9c28ac3dc8b" (UID: "173401a9-067f-4b30-b2e4-f9c28ac3dc8b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:32.942651 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:32.942586    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-rbac-proxy-sar-config" pod "173401a9-067f-4b30-b2e4-f9c28ac3dc8b" (UID: "173401a9-067f-4b30-b2e4-f9c28ac3dc8b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:32.944573 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:32.944545    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-proxy-tls" pod "173401a9-067f-4b30-b2e4-f9c28ac3dc8b" (UID: "173401a9-067f-4b30-b2e4-f9c28ac3dc8b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:32.944664 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:32.944566    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-api-access-ln546" pod "173401a9-067f-4b30-b2e4-f9c28ac3dc8b" (UID: "173401a9-067f-4b30-b2e4-f9c28ac3dc8b"). InnerVolumeSpecName "kube-api-access-ln546". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:33.043192 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.043157    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln546\" (UniqueName: \"kubernetes.io/projected/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-api-access-ln546\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:56:33.043192 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.043189    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:56:33.043399 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.043205    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:56:33.043399 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.043218    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173401a9-067f-4b30-b2e4-f9c28ac3dc8b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:56:33.290012 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.289984    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn_173401a9-067f-4b30-b2e4-f9c28ac3dc8b/storage-initializer/1.log"
Oct 30 03:56:33.290401 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.290386    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn_173401a9-067f-4b30-b2e4-f9c28ac3dc8b/storage-initializer/0.log"
Oct 30 03:56:33.290476 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.290426    4885 generic.go:356] "Generic (PLEG): container finished" podID="173401a9-067f-4b30-b2e4-f9c28ac3dc8b" containerID="4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf" exitCode=1
Oct 30 03:56:33.290523 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.290494    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"
Oct 30 03:56:33.290523 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.290498    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn" event={"ID":"173401a9-067f-4b30-b2e4-f9c28ac3dc8b","Type":"ContainerDied","Data":"4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf"}
Oct 30 03:56:33.290588 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.290559    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn" event={"ID":"173401a9-067f-4b30-b2e4-f9c28ac3dc8b","Type":"ContainerDied","Data":"4709e58508af664556a785aee7ceb82ff07c3f14d35994ea658a62fe53b398a5"}
Oct 30 03:56:33.290588 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.290586    4885 scope.go:122] "RemoveContainer" containerID="4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf"
Oct 30 03:56:33.302124 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.301779    4885 scope.go:122] "RemoveContainer" containerID="27cc691f931c09c11a8678f86b282824cf9212d3694975881c7d60c379f7ea70"
Oct 30 03:56:33.317820 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.317795    4885 scope.go:122] "RemoveContainer" containerID="4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf"
Oct 30 03:56:33.318375 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:56:33.318349    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf\": container with ID starting with 4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf not found: ID does not exist" containerID="4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf"
Oct 30 03:56:33.318519 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.318495    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf"} err="failed to get container status \"4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf\": rpc error: code = NotFound desc = could not find container \"4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf\": container with ID starting with 4dbc9d7815784e5e21566c484057007e0d0f60bf60fd07a8b51289d9787200bf not found: ID does not exist"
Oct 30 03:56:33.318632 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.318603    4885 scope.go:122] "RemoveContainer" containerID="27cc691f931c09c11a8678f86b282824cf9212d3694975881c7d60c379f7ea70"
Oct 30 03:56:33.319077 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:56:33.319044    4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27cc691f931c09c11a8678f86b282824cf9212d3694975881c7d60c379f7ea70\": container with ID starting with 27cc691f931c09c11a8678f86b282824cf9212d3694975881c7d60c379f7ea70 not found: ID does not exist" containerID="27cc691f931c09c11a8678f86b282824cf9212d3694975881c7d60c379f7ea70"
Oct 30 03:56:33.319160 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.319086    4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27cc691f931c09c11a8678f86b282824cf9212d3694975881c7d60c379f7ea70"} err="failed to get container status \"27cc691f931c09c11a8678f86b282824cf9212d3694975881c7d60c379f7ea70\": rpc error: code = NotFound desc = could not find container \"27cc691f931c09c11a8678f86b282824cf9212d3694975881c7d60c379f7ea70\": container with ID starting with 27cc691f931c09c11a8678f86b282824cf9212d3694975881c7d60c379f7ea70 not found: ID does not exist"
Oct 30 03:56:33.367757 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.367713    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"]
Oct 30 03:56:33.371836 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:33.371794    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-fail-predictor-54565697c8-b8rcn"]
Oct 30 03:56:34.628966 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:34.628931    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="173401a9-067f-4b30-b2e4-f9c28ac3dc8b" path="/var/lib/kubelet/pods/173401a9-067f-4b30-b2e4-f9c28ac3dc8b/volumes"
Oct 30 03:56:35.302831 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:35.302786    4885 generic.go:356] "Generic (PLEG): container finished" podID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerID="08020f4c35d79aceb97e24c0d9bb049fa987bc6903dff520a2a5a1d1ccd8d3fb" exitCode=0
Oct 30 03:56:35.303037 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:35.302859    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz" event={"ID":"09b243f7-4c85-4198-b1ac-1aaae8bab7e7","Type":"ContainerDied","Data":"08020f4c35d79aceb97e24c0d9bb049fa987bc6903dff520a2a5a1d1ccd8d3fb"}
Oct 30 03:56:36.310301 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:36.310259    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz" event={"ID":"09b243f7-4c85-4198-b1ac-1aaae8bab7e7","Type":"ContainerStarted","Data":"1cc0a9355421fdd7c17ea3afb7e07705a54bba5aa50dc4ee8c960247583a9e91"}
Oct 30 03:56:36.310732 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:36.310344    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz" event={"ID":"09b243f7-4c85-4198-b1ac-1aaae8bab7e7","Type":"ContainerStarted","Data":"7c218350eec44a37854e7225c7b4c107375efc8f343e1e1ff2b69eac7559e117"}
Oct 30 03:56:36.310732 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:36.310566    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:36.374701 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:36.374650    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz" podStartSLOduration=2.058056833 podStartE2EDuration="6.374635959s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.316579126s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:36.373762994 +0000 UTC m=+3701.883932021" watchObservedRunningTime="2026-10-30 03:56:36.374635959 +0000 UTC m=+3701.884804981"
Oct 30 03:56:37.315180 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:37.315148    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:43.324651 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:56:43.324603    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:56:55.921565 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:56:55.921527    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:57:13.329261 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:13.329181    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:57:20.269453 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:20.269418    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"]
Oct 30 03:57:20.269975 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:20.269732    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz" podUID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerName="kserve-container" containerID="cri-o://7c218350eec44a37854e7225c7b4c107375efc8f343e1e1ff2b69eac7559e117" gracePeriod=30
Oct 30 03:57:20.269975 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:20.269764    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz" podUID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerName="kube-rbac-proxy" containerID="cri-o://1cc0a9355421fdd7c17ea3afb7e07705a54bba5aa50dc4ee8c960247583a9e91" gracePeriod=30
Oct 30 03:57:20.499534 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:20.499497    4885 generic.go:356] "Generic (PLEG): container finished" podID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerID="1cc0a9355421fdd7c17ea3afb7e07705a54bba5aa50dc4ee8c960247583a9e91" exitCode=0
Oct 30 03:57:20.499741 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:20.499556    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz" event={"ID":"09b243f7-4c85-4198-b1ac-1aaae8bab7e7","Type":"ContainerDied","Data":"1cc0a9355421fdd7c17ea3afb7e07705a54bba5aa50dc4ee8c960247583a9e91"}
Oct 30 03:57:21.506015 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.505987    4885 generic.go:356] "Generic (PLEG): container finished" podID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerID="7c218350eec44a37854e7225c7b4c107375efc8f343e1e1ff2b69eac7559e117" exitCode=0
Oct 30 03:57:21.506389 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.506039    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz" event={"ID":"09b243f7-4c85-4198-b1ac-1aaae8bab7e7","Type":"ContainerDied","Data":"7c218350eec44a37854e7225c7b4c107375efc8f343e1e1ff2b69eac7559e117"}
Oct 30 03:57:21.506389 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.506156    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz" event={"ID":"09b243f7-4c85-4198-b1ac-1aaae8bab7e7","Type":"ContainerDied","Data":"53c5f4f28c0079a203a6c7af1de616727779134817f123c5c7a34b3044e3568e"}
Oct 30 03:57:21.506389 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.506190    4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53c5f4f28c0079a203a6c7af1de616727779134817f123c5c7a34b3044e3568e"
Oct 30 03:57:21.517280 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.517256    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:57:21.639077 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.639002    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-proxy-tls\") pod \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") "
Oct 30 03:57:21.639077 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.639044    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-rbac-proxy-sar-config\") pod \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") "
Oct 30 03:57:21.639324 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.639084    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-api-access-msbvm\" (UniqueName: \"kubernetes.io/projected/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-api-access-msbvm\") pod \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") "
Oct 30 03:57:21.639324 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.639128    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kserve-provision-location\") pod \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") "
Oct 30 03:57:21.639324 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.639149    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-cabundle-cert\") pod \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\" (UID: \"09b243f7-4c85-4198-b1ac-1aaae8bab7e7\") "
Oct 30 03:57:21.639592 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.639562    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-rbac-proxy-sar-config" pod "09b243f7-4c85-4198-b1ac-1aaae8bab7e7" (UID: "09b243f7-4c85-4198-b1ac-1aaae8bab7e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:21.639592 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.639572    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kserve-provision-location" pod "09b243f7-4c85-4198-b1ac-1aaae8bab7e7" (UID: "09b243f7-4c85-4198-b1ac-1aaae8bab7e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:21.639592 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.639587    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-cabundle-cert" pod "09b243f7-4c85-4198-b1ac-1aaae8bab7e7" (UID: "09b243f7-4c85-4198-b1ac-1aaae8bab7e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:21.641339 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.641317    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-proxy-tls" pod "09b243f7-4c85-4198-b1ac-1aaae8bab7e7" (UID: "09b243f7-4c85-4198-b1ac-1aaae8bab7e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:21.641339 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.641335    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-api-access-msbvm" pod "09b243f7-4c85-4198-b1ac-1aaae8bab7e7" (UID: "09b243f7-4c85-4198-b1ac-1aaae8bab7e7"). InnerVolumeSpecName "kube-api-access-msbvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:21.740568 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.740525    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:57:21.740568 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.740564    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:57:21.740568 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.740579    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msbvm\" (UniqueName: \"kubernetes.io/projected/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kube-api-access-msbvm\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:57:21.740837 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.740592    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:57:21.740837 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:21.740606    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09b243f7-4c85-4198-b1ac-1aaae8bab7e7-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:57:22.510145 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:22.510115    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"
Oct 30 03:57:22.566008 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:22.565948    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"]
Oct 30 03:57:22.575814 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:22.575772    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-mlflow-v2-runtime-predictor-68cb8fc678-rnfwz"]
Oct 30 03:57:22.623858 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:22.623829    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" path="/var/lib/kubelet/pods/09b243f7-4c85-4198-b1ac-1aaae8bab7e7/volumes"
Oct 30 03:57:23.822005 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.821941    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"]
Oct 30 03:57:23.822543 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822448    4885 cpu_manager.go:414] "containerMap: removing container" podUID="173401a9-067f-4b30-b2e4-f9c28ac3dc8b" containerName="storage-initializer"
Oct 30 03:57:23.822543 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822488    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="173401a9-067f-4b30-b2e4-f9c28ac3dc8b" containerName="storage-initializer"
Oct 30 03:57:23.822543 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822510    4885 cpu_manager.go:414] "containerMap: removing container" podUID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerName="kserve-container"
Oct 30 03:57:23.822543 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822518    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerName="kserve-container"
Oct 30 03:57:23.822543 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822529    4885 cpu_manager.go:414] "containerMap: removing container" podUID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerName="kube-rbac-proxy"
Oct 30 03:57:23.822543 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822535    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerName="kube-rbac-proxy"
Oct 30 03:57:23.822887 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822564    4885 cpu_manager.go:414] "containerMap: removing container" podUID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerName="storage-initializer"
Oct 30 03:57:23.822887 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822569    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerName="storage-initializer"
Oct 30 03:57:23.822887 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822660    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="173401a9-067f-4b30-b2e4-f9c28ac3dc8b" containerName="storage-initializer"
Oct 30 03:57:23.822887 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822671    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerName="kserve-container"
Oct 30 03:57:23.822887 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822677    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b243f7-4c85-4198-b1ac-1aaae8bab7e7" containerName="kube-rbac-proxy"
Oct 30 03:57:23.822887 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822766    4885 cpu_manager.go:414] "containerMap: removing container" podUID="173401a9-067f-4b30-b2e4-f9c28ac3dc8b" containerName="storage-initializer"
Oct 30 03:57:23.822887 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822773    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="173401a9-067f-4b30-b2e4-f9c28ac3dc8b" containerName="storage-initializer"
Oct 30 03:57:23.822887 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.822835    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="173401a9-067f-4b30-b2e4-f9c28ac3dc8b" containerName="storage-initializer"
Oct 30 03:57:23.827097 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.827081    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:23.839994 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.839970    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Oct 30 03:57:23.849838 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.849814    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:57:23.849985 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.849860    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:57:23.849985 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.849860    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Oct 30 03:57:23.849985 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.849875    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:57:23.849985 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.849866    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:57:23.850197 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.850055    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:57:23.851526 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.851504    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"]
Oct 30 03:57:23.957878 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.957838    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:23.958069 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.957886    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:23.958069 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.957973    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:23.958069 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.958030    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:23.958204 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:23.958094    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfhw\" (UniqueName: \"kubernetes.io/projected/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-api-access-sqfhw\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:24.059406 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.059361    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:24.059658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.059427    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfhw\" (UniqueName: \"kubernetes.io/projected/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-api-access-sqfhw\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:24.059658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.059472    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:24.059658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.059500    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:24.059658 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.059529    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:24.059996 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.059959    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:24.060157 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.060135    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:24.060227 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.060197    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:24.062143 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.062128    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:24.081086 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.081017    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfhw\" (UniqueName: \"kubernetes.io/projected/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-api-access-sqfhw\") pod \"isvc-sklearn-mcp-predictor-5d8d6d494-95zhd\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") " pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:24.146708 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.146677    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:24.464733 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.464699    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"]
Oct 30 03:57:24.468031 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:57:24.468003    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e953ae1_148d_4270_9e3f_8edcbf5d7a8f.slice/crio-dbd72f39bd6caff2e90b01b355dec4fbbd987653b3eee4acd36b74d1b2dacef2 WatchSource:0}: Error finding container dbd72f39bd6caff2e90b01b355dec4fbbd987653b3eee4acd36b74d1b2dacef2: Status 404 returned error can't find the container with id dbd72f39bd6caff2e90b01b355dec4fbbd987653b3eee4acd36b74d1b2dacef2
Oct 30 03:57:24.520671 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:24.520642    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" event={"ID":"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f","Type":"ContainerStarted","Data":"dbd72f39bd6caff2e90b01b355dec4fbbd987653b3eee4acd36b74d1b2dacef2"}
Oct 30 03:57:25.526659 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:25.526627    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" event={"ID":"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f","Type":"ContainerStarted","Data":"8664d9e7c89d2f8bc49eb20d7d6604e97b713b482cd40823d3cf520f1a2bb7fc"}
Oct 30 03:57:25.921699 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:57:25.921598    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:57:28.540348 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:28.540312    4885 generic.go:356] "Generic (PLEG): container finished" podID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerID="8664d9e7c89d2f8bc49eb20d7d6604e97b713b482cd40823d3cf520f1a2bb7fc" exitCode=0
Oct 30 03:57:28.540348 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:28.540351    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" event={"ID":"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f","Type":"ContainerDied","Data":"8664d9e7c89d2f8bc49eb20d7d6604e97b713b482cd40823d3cf520f1a2bb7fc"}
Oct 30 03:57:29.550311 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:29.550271    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" event={"ID":"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f","Type":"ContainerStarted","Data":"515bbd74e1e15c9596a0b8f0a46beca64f52f7244a4ee81780df96923f1c703e"}
Oct 30 03:57:31.562739 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:31.562706    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" event={"ID":"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f","Type":"ContainerStarted","Data":"4afd760a69ce0ce0139154c1111a32ede417180084eb238deac2257979811867"}
Oct 30 03:57:31.563109 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:31.562766    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" event={"ID":"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f","Type":"ContainerStarted","Data":"24229c1b44cae1c02d15b95f45c223ccc200a5a824e14a2a6666eaf44101eca5"}
Oct 30 03:57:31.563109 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:31.562838    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:31.563109 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:31.562901    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:31.563109 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:31.562952    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:31.626247 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:31.626195    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podStartSLOduration=1.962964082 podStartE2EDuration="8.626178976s" totalImagesPullingTime="2.780967747s" totalInitContainerRuntime="3.882247147s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:31.614997268 +0000 UTC m=+3757.125166313" watchObservedRunningTime="2026-10-30 03:57:31.626178976 +0000 UTC m=+3757.136348005"
Oct 30 03:57:37.572801 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:57:37.572775    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:57:55.921949 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:57:55.921909    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:58:04.148230 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:04.148177    4885 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.93:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.93:8080: connect: connection refused"
Oct 30 03:58:07.742829 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:07.742799    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:58:25.922521 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:58:25.922477    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:58:37.744072 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:37.743988    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:58:43.974083 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:43.974050    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"]
Oct 30 03:58:43.974507 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:43.974308    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kserve-container" containerID="cri-o://515bbd74e1e15c9596a0b8f0a46beca64f52f7244a4ee81780df96923f1c703e" gracePeriod=30
Oct 30 03:58:43.974507 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:43.974347    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kserve-agent" containerID="cri-o://24229c1b44cae1c02d15b95f45c223ccc200a5a824e14a2a6666eaf44101eca5" gracePeriod=30
Oct 30 03:58:43.974507 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:43.974350    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kube-rbac-proxy" containerID="cri-o://4afd760a69ce0ce0139154c1111a32ede417180084eb238deac2257979811867" gracePeriod=30
Oct 30 03:58:44.907980 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:44.907946    4885 generic.go:356] "Generic (PLEG): container finished" podID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerID="4afd760a69ce0ce0139154c1111a32ede417180084eb238deac2257979811867" exitCode=0
Oct 30 03:58:44.908168 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:44.908017    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" event={"ID":"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f","Type":"ContainerDied","Data":"4afd760a69ce0ce0139154c1111a32ede417180084eb238deac2257979811867"}
Oct 30 03:58:46.918834 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:46.918803    4885 generic.go:356] "Generic (PLEG): container finished" podID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerID="515bbd74e1e15c9596a0b8f0a46beca64f52f7244a4ee81780df96923f1c703e" exitCode=0
Oct 30 03:58:46.919218 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:46.918879    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" event={"ID":"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f","Type":"ContainerDied","Data":"515bbd74e1e15c9596a0b8f0a46beca64f52f7244a4ee81780df96923f1c703e"}
Oct 30 03:58:47.568020 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:47.567979    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.93:8643/healthz\": dial tcp 10.132.0.93:8643: connect: connection refused"
Oct 30 03:58:47.573653 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:47.573605    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.93:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.93:8080: connect: connection refused"
Oct 30 03:58:52.567988 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:52.567948    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.93:8643/healthz\": dial tcp 10.132.0.93:8643: connect: connection refused"
Oct 30 03:58:55.921772 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:58:55.921733    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:58:57.567865 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:57.567822    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.93:8643/healthz\": dial tcp 10.132.0.93:8643: connect: connection refused"
Oct 30 03:58:57.568344 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:57.567986    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:58:57.573485 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:58:57.573452    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.93:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.93:8080: connect: connection refused"
Oct 30 03:59:02.567466 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:02.567422    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.93:8643/healthz\": dial tcp 10.132.0.93:8643: connect: connection refused"
Oct 30 03:59:07.568246 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:07.568204    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.93:8643/healthz\": dial tcp 10.132.0.93:8643: connect: connection refused"
Oct 30 03:59:07.573925 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:07.573894    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.93:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.93:8080: connect: connection refused"
Oct 30 03:59:07.574042 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:07.574031    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:59:12.567862 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:12.567818    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.93:8643/healthz\": dial tcp 10.132.0.93:8643: connect: connection refused"
Oct 30 03:59:14.041194 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.041159    4885 generic.go:356] "Generic (PLEG): container finished" podID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerID="24229c1b44cae1c02d15b95f45c223ccc200a5a824e14a2a6666eaf44101eca5" exitCode=137
Oct 30 03:59:14.041552 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.041226    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" event={"ID":"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f","Type":"ContainerDied","Data":"24229c1b44cae1c02d15b95f45c223ccc200a5a824e14a2a6666eaf44101eca5"}
Oct 30 03:59:14.126129 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.126108    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:59:14.157303 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.157275    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kserve-provision-location\") pod \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") "
Oct 30 03:59:14.157303 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.157306    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-cabundle-cert\") pod \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") "
Oct 30 03:59:14.157464 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.157373    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-rbac-proxy-sar-config\") pod \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") "
Oct 30 03:59:14.157464 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.157407    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-api-access-sqfhw\" (UniqueName: \"kubernetes.io/projected/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-api-access-sqfhw\") pod \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") "
Oct 30 03:59:14.157464 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.157456    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-proxy-tls\") pod \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\" (UID: \"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f\") "
Oct 30 03:59:14.157662 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.157636    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kserve-provision-location" pod "9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" (UID: "9e953ae1-148d-4270-9e3f-8edcbf5d7a8f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:14.157736 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.157717    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-cabundle-cert" pod "9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" (UID: "9e953ae1-148d-4270-9e3f-8edcbf5d7a8f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:14.157801 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.157750    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-rbac-proxy-sar-config" pod "9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" (UID: "9e953ae1-148d-4270-9e3f-8edcbf5d7a8f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:14.159595 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.159570    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-api-access-sqfhw" pod "9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" (UID: "9e953ae1-148d-4270-9e3f-8edcbf5d7a8f"). InnerVolumeSpecName "kube-api-access-sqfhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:59:14.159706 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.159670    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-proxy-tls" pod "9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" (UID: "9e953ae1-148d-4270-9e3f-8edcbf5d7a8f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:59:14.258208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.258126    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:59:14.258208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.258158    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqfhw\" (UniqueName: \"kubernetes.io/projected/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kube-api-access-sqfhw\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:59:14.258208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.258172    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:59:14.258208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.258184    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:59:14.258208 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:14.258197    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 03:59:15.048085 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:15.048059    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"
Oct 30 03:59:15.048085 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:15.048066    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd" event={"ID":"9e953ae1-148d-4270-9e3f-8edcbf5d7a8f","Type":"ContainerDied","Data":"dbd72f39bd6caff2e90b01b355dec4fbbd987653b3eee4acd36b74d1b2dacef2"}
Oct 30 03:59:15.048559 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:15.048139    4885 scope.go:122] "RemoveContainer" containerID="4afd760a69ce0ce0139154c1111a32ede417180084eb238deac2257979811867"
Oct 30 03:59:15.056651 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:15.056607    4885 scope.go:122] "RemoveContainer" containerID="24229c1b44cae1c02d15b95f45c223ccc200a5a824e14a2a6666eaf44101eca5"
Oct 30 03:59:15.064446 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:15.064426    4885 scope.go:122] "RemoveContainer" containerID="515bbd74e1e15c9596a0b8f0a46beca64f52f7244a4ee81780df96923f1c703e"
Oct 30 03:59:15.071728 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:15.071708    4885 scope.go:122] "RemoveContainer" containerID="8664d9e7c89d2f8bc49eb20d7d6604e97b713b482cd40823d3cf520f1a2bb7fc"
Oct 30 03:59:15.104402 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:15.104375    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"]
Oct 30 03:59:15.114788 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:15.114763    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-sklearn-mcp-predictor-5d8d6d494-95zhd"]
Oct 30 03:59:16.116897 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.116856    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"]
Oct 30 03:59:16.117395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.117296    4885 cpu_manager.go:414] "containerMap: removing container" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kserve-container"
Oct 30 03:59:16.117395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.117315    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kserve-container"
Oct 30 03:59:16.117395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.117335    4885 cpu_manager.go:414] "containerMap: removing container" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kube-rbac-proxy"
Oct 30 03:59:16.117395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.117342    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kube-rbac-proxy"
Oct 30 03:59:16.117395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.117361    4885 cpu_manager.go:414] "containerMap: removing container" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="storage-initializer"
Oct 30 03:59:16.117395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.117368    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="storage-initializer"
Oct 30 03:59:16.117395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.117388    4885 cpu_manager.go:414] "containerMap: removing container" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kserve-agent"
Oct 30 03:59:16.117395 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.117398    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kserve-agent"
Oct 30 03:59:16.117853 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.117482    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kserve-container"
Oct 30 03:59:16.117853 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.117495    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kube-rbac-proxy"
Oct 30 03:59:16.117853 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.117509    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" containerName="kserve-agent"
Oct 30 03:59:16.122207 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.122186    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.135375 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.135352    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Oct 30 03:59:16.135496 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.135396    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:59:16.135496 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.135404    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:59:16.145425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.145403    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:59:16.145425 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.145404    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:59:16.145569 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.145426    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:59:16.145569 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.145437    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-paddle-predictor-serving-cert\""
Oct 30 03:59:16.146744 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.146723    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"]
Oct 30 03:59:16.171356 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.171332    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtl5b\" (UniqueName: \"kubernetes.io/projected/b4873e55-2083-4a06-9720-9333f216a88d-kube-api-access-xtl5b\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.171450 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.171379    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4873e55-2083-4a06-9720-9333f216a88d-kserve-provision-location\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.171498 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.171450    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4873e55-2083-4a06-9720-9333f216a88d-proxy-tls\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.171498 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.171483    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.171574 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.171507    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-cabundle-cert\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.272519 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.272472    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.272519 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.272511    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-cabundle-cert\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.272817 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.272564    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtl5b\" (UniqueName: \"kubernetes.io/projected/b4873e55-2083-4a06-9720-9333f216a88d-kube-api-access-xtl5b\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.272817 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.272603    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4873e55-2083-4a06-9720-9333f216a88d-kserve-provision-location\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.272817 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.272661    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4873e55-2083-4a06-9720-9333f216a88d-proxy-tls\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.273104 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.273075    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4873e55-2083-4a06-9720-9333f216a88d-kserve-provision-location\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.273291 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.273269    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-cabundle-cert\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.273356 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.273270    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.275198 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.275174    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4873e55-2083-4a06-9720-9333f216a88d-proxy-tls\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.294933 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.294901    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtl5b\" (UniqueName: \"kubernetes.io/projected/b4873e55-2083-4a06-9720-9333f216a88d-kube-api-access-xtl5b\") pod \"isvc-paddle-predictor-ff9f74dd7-wk2jm\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.439549 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.439461    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:16.623531 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.623501    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e953ae1-148d-4270-9e3f-8edcbf5d7a8f" path="/var/lib/kubelet/pods/9e953ae1-148d-4270-9e3f-8edcbf5d7a8f/volumes"
Oct 30 03:59:16.745412 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:16.745382    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"]
Oct 30 03:59:16.748517 ip-10-0-140-132 kubenswrapper[4885]: W1030 03:59:16.748490    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4873e55_2083_4a06_9720_9333f216a88d.slice/crio-fdc52c80fb2002e0425a933d9b4919cd45bf2bd3dc5763db8e42f5f4c47f8b71 WatchSource:0}: Error finding container fdc52c80fb2002e0425a933d9b4919cd45bf2bd3dc5763db8e42f5f4c47f8b71: Status 404 returned error can't find the container with id fdc52c80fb2002e0425a933d9b4919cd45bf2bd3dc5763db8e42f5f4c47f8b71
Oct 30 03:59:17.059302 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:17.059210    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" event={"ID":"b4873e55-2083-4a06-9720-9333f216a88d","Type":"ContainerStarted","Data":"fcfcf8f28e4918a9072018768af3755f432114e62d9f2548b17da82f46f038cd"}
Oct 30 03:59:17.059302 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:17.059287    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" event={"ID":"b4873e55-2083-4a06-9720-9333f216a88d","Type":"ContainerStarted","Data":"fdc52c80fb2002e0425a933d9b4919cd45bf2bd3dc5763db8e42f5f4c47f8b71"}
Oct 30 03:59:22.083467 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:22.083436    4885 generic.go:356] "Generic (PLEG): container finished" podID="b4873e55-2083-4a06-9720-9333f216a88d" containerID="fcfcf8f28e4918a9072018768af3755f432114e62d9f2548b17da82f46f038cd" exitCode=0
Oct 30 03:59:22.083944 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:22.083491    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" event={"ID":"b4873e55-2083-4a06-9720-9333f216a88d","Type":"ContainerDied","Data":"fcfcf8f28e4918a9072018768af3755f432114e62d9f2548b17da82f46f038cd"}
Oct 30 03:59:25.925797 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:59:25.925755    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 03:59:34.148476 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:34.148438    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" event={"ID":"b4873e55-2083-4a06-9720-9333f216a88d","Type":"ContainerStarted","Data":"6f738bf786d6d0ba087702adddc20dff25e2b67b891b8e26c2466456592b642a"}
Oct 30 03:59:34.148953 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:34.148530    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" event={"ID":"b4873e55-2083-4a06-9720-9333f216a88d","Type":"ContainerStarted","Data":"518799ae9fa5ad92da7db9db7bbc12cf5c796c1ca06a0f551d4f2b40eda12944"}
Oct 30 03:59:34.148953 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:34.148802    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:34.148953 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:34.148850    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:34.150108 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:34.150084    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.94:8080: connect: connection refused"
Oct 30 03:59:34.189806 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:34.189761    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" podStartSLOduration=2.347591374 podStartE2EDuration="18.189748489s" totalImagesPullingTime="11.228111422s" totalInitContainerRuntime="4.614045693s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:59:34.189531505 +0000 UTC m=+3879.699700532" watchObservedRunningTime="2026-10-30 03:59:34.189748489 +0000 UTC m=+3879.699917511"
Oct 30 03:59:35.152717 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:35.152679    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.94:8080: connect: connection refused"
Oct 30 03:59:40.157412 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:40.157381    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 03:59:40.157935 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:40.157908    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.94:8080: connect: connection refused"
Oct 30 03:59:50.158857 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:50.158816    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.94:8080: connect: connection refused"
Oct 30 03:59:55.010948 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:55.010822    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:59:55.020710 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:55.015536    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 03:59:55.020710 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:55.016592    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:59:55.020937 ip-10-0-140-132 kubenswrapper[4885]: I1030 03:59:55.020921    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244kd_971e53b3-ff90-4be0-bf64-9c1013099b9d/ovn-acl-logging/0.log"
Oct 30 03:59:55.922220 ip-10-0-140-132 kubenswrapper[4885]: E1030 03:59:55.922183    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 04:00:00.158753 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:00.158717    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.94:8080: connect: connection refused"
Oct 30 04:00:10.158345 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:10.158253    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.94:8080: connect: connection refused"
Oct 30 04:00:20.158741 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:20.158706    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 04:00:25.921134 ip-10-0-140-132 kubenswrapper[4885]: E1030 04:00:25.921101    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 04:00:27.658066 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:27.658033    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"]
Oct 30 04:00:27.658487 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:27.658305    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kserve-container" containerID="cri-o://518799ae9fa5ad92da7db9db7bbc12cf5c796c1ca06a0f551d4f2b40eda12944" gracePeriod=30
Oct 30 04:00:27.658586 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:27.658461    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kube-rbac-proxy" containerID="cri-o://6f738bf786d6d0ba087702adddc20dff25e2b67b891b8e26c2466456592b642a" gracePeriod=30
Oct 30 04:00:28.377907 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:28.377878    4885 generic.go:356] "Generic (PLEG): container finished" podID="b4873e55-2083-4a06-9720-9333f216a88d" containerID="6f738bf786d6d0ba087702adddc20dff25e2b67b891b8e26c2466456592b642a" exitCode=0
Oct 30 04:00:28.378080 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:28.377953    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" event={"ID":"b4873e55-2083-4a06-9720-9333f216a88d","Type":"ContainerDied","Data":"6f738bf786d6d0ba087702adddc20dff25e2b67b891b8e26c2466456592b642a"}
Oct 30 04:00:30.153857 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.153815    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.94:8643/healthz\": dial tcp 10.132.0.94:8643: connect: connection refused"
Oct 30 04:00:30.158178 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.158150    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.94:8080: connect: connection refused"
Oct 30 04:00:30.389200 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.389174    4885 generic.go:356] "Generic (PLEG): container finished" podID="b4873e55-2083-4a06-9720-9333f216a88d" containerID="518799ae9fa5ad92da7db9db7bbc12cf5c796c1ca06a0f551d4f2b40eda12944" exitCode=0
Oct 30 04:00:30.389310 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.389248    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" event={"ID":"b4873e55-2083-4a06-9720-9333f216a88d","Type":"ContainerDied","Data":"518799ae9fa5ad92da7db9db7bbc12cf5c796c1ca06a0f551d4f2b40eda12944"}
Oct 30 04:00:30.499488 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.499465    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 04:00:30.588235 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.588200    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4873e55-2083-4a06-9720-9333f216a88d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4873e55-2083-4a06-9720-9333f216a88d-proxy-tls\") pod \"b4873e55-2083-4a06-9720-9333f216a88d\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") "
Oct 30 04:00:30.588411 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.588246    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4873e55-2083-4a06-9720-9333f216a88d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4873e55-2083-4a06-9720-9333f216a88d-kserve-provision-location\") pod \"b4873e55-2083-4a06-9720-9333f216a88d\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") "
Oct 30 04:00:30.588411 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.588287    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-kube-rbac-proxy-sar-config\") pod \"b4873e55-2083-4a06-9720-9333f216a88d\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") "
Oct 30 04:00:30.588411 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.588308    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4873e55-2083-4a06-9720-9333f216a88d-kube-api-access-xtl5b\" (UniqueName: \"kubernetes.io/projected/b4873e55-2083-4a06-9720-9333f216a88d-kube-api-access-xtl5b\") pod \"b4873e55-2083-4a06-9720-9333f216a88d\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") "
Oct 30 04:00:30.588411 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.588344    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-cabundle-cert\") pod \"b4873e55-2083-4a06-9720-9333f216a88d\" (UID: \"b4873e55-2083-4a06-9720-9333f216a88d\") "
Oct 30 04:00:30.588801 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.588751    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-kube-rbac-proxy-sar-config" pod "b4873e55-2083-4a06-9720-9333f216a88d" (UID: "b4873e55-2083-4a06-9720-9333f216a88d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:30.588900 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.588848    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-cabundle-cert" pod "b4873e55-2083-4a06-9720-9333f216a88d" (UID: "b4873e55-2083-4a06-9720-9333f216a88d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:30.590547 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.590523    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4873e55-2083-4a06-9720-9333f216a88d-kube-api-access-xtl5b" pod "b4873e55-2083-4a06-9720-9333f216a88d" (UID: "b4873e55-2083-4a06-9720-9333f216a88d"). InnerVolumeSpecName "kube-api-access-xtl5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:30.590605 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.590554    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4873e55-2083-4a06-9720-9333f216a88d-proxy-tls" pod "b4873e55-2083-4a06-9720-9333f216a88d" (UID: "b4873e55-2083-4a06-9720-9333f216a88d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:00:30.596774 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.596753    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4873e55-2083-4a06-9720-9333f216a88d-kserve-provision-location" pod "b4873e55-2083-4a06-9720-9333f216a88d" (UID: "b4873e55-2083-4a06-9720-9333f216a88d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:30.689544 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.689451    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4873e55-2083-4a06-9720-9333f216a88d-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 04:00:30.689544 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.689491    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4873e55-2083-4a06-9720-9333f216a88d-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 04:00:30.689544 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.689507    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 04:00:30.689544 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.689521    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtl5b\" (UniqueName: \"kubernetes.io/projected/b4873e55-2083-4a06-9720-9333f216a88d-kube-api-access-xtl5b\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 04:00:30.689544 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:30.689534    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4873e55-2083-4a06-9720-9333f216a88d-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 04:00:31.396369 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:31.396332    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm" event={"ID":"b4873e55-2083-4a06-9720-9333f216a88d","Type":"ContainerDied","Data":"fdc52c80fb2002e0425a933d9b4919cd45bf2bd3dc5763db8e42f5f4c47f8b71"}
Oct 30 04:00:31.396369 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:31.396362    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"
Oct 30 04:00:31.396960 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:31.396399    4885 scope.go:122] "RemoveContainer" containerID="6f738bf786d6d0ba087702adddc20dff25e2b67b891b8e26c2466456592b642a"
Oct 30 04:00:31.405170 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:31.405145    4885 scope.go:122] "RemoveContainer" containerID="518799ae9fa5ad92da7db9db7bbc12cf5c796c1ca06a0f551d4f2b40eda12944"
Oct 30 04:00:31.412933 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:31.412913    4885 scope.go:122] "RemoveContainer" containerID="fcfcf8f28e4918a9072018768af3755f432114e62d9f2548b17da82f46f038cd"
Oct 30 04:00:31.453066 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:31.453038    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"]
Oct 30 04:00:31.458721 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:31.458703    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-paddle-predictor-ff9f74dd7-wk2jm"]
Oct 30 04:00:32.624097 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:32.624068    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4873e55-2083-4a06-9720-9333f216a88d" path="/var/lib/kubelet/pods/b4873e55-2083-4a06-9720-9333f216a88d/volumes"
Oct 30 04:00:33.208382 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.208345    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"]
Oct 30 04:00:33.208759 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.208739    4885 cpu_manager.go:414] "containerMap: removing container" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="storage-initializer"
Oct 30 04:00:33.208883 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.208761    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="storage-initializer"
Oct 30 04:00:33.208883 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.208799    4885 cpu_manager.go:414] "containerMap: removing container" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kserve-container"
Oct 30 04:00:33.208883 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.208807    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kserve-container"
Oct 30 04:00:33.208883 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.208826    4885 cpu_manager.go:414] "containerMap: removing container" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kube-rbac-proxy"
Oct 30 04:00:33.208883 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.208834    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kube-rbac-proxy"
Oct 30 04:00:33.209144 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.208924    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kserve-container"
Oct 30 04:00:33.209144 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.208939    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4873e55-2083-4a06-9720-9333f216a88d" containerName="kube-rbac-proxy"
Oct 30 04:00:33.211892 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.211870    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.224918 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.224891    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Oct 30 04:00:33.225023 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.224895    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 04:00:33.235027 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.235009    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 04:00:33.235286 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.235270    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Oct 30 04:00:33.235359 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.235270    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:00:33.235359 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.235335    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 04:00:33.235359 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.235356    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 04:00:33.238436 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.238415    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"]
Oct 30 04:00:33.308627 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.308573    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1064998-e955-4455-8402-bc240e20b88b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.308796 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.308638    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.308796 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.308733    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.308796 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.308763    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1064998-e955-4455-8402-bc240e20b88b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.308962 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.308800    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8m6d\" (UniqueName: \"kubernetes.io/projected/e1064998-e955-4455-8402-bc240e20b88b-kube-api-access-k8m6d\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.409163 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.409130    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.409341 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.409171    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.409341 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.409193    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1064998-e955-4455-8402-bc240e20b88b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.409341 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.409216    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8m6d\" (UniqueName: \"kubernetes.io/projected/e1064998-e955-4455-8402-bc240e20b88b-kube-api-access-k8m6d\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.409341 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.409281    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1064998-e955-4455-8402-bc240e20b88b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.409721 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.409695    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1064998-e955-4455-8402-bc240e20b88b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.409887 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.409868    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.410001 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.409977    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.411924 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.411905    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1064998-e955-4455-8402-bc240e20b88b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.430943 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.430911    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8m6d\" (UniqueName: \"kubernetes.io/projected/e1064998-e955-4455-8402-bc240e20b88b-kube-api-access-k8m6d\") pod \"isvc-paddle-runtime-predictor-5488b85b79-6ljz9\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") " pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.527632 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.527549    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:33.837453 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:33.837424    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"]
Oct 30 04:00:33.840286 ip-10-0-140-132 kubenswrapper[4885]: W1030 04:00:33.840263    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1064998_e955_4455_8402_bc240e20b88b.slice/crio-0ee685cd1f7afc5c21af9984947678d9ad2fa3bab429b5611e86108d942f3c8a WatchSource:0}: Error finding container 0ee685cd1f7afc5c21af9984947678d9ad2fa3bab429b5611e86108d942f3c8a: Status 404 returned error can't find the container with id 0ee685cd1f7afc5c21af9984947678d9ad2fa3bab429b5611e86108d942f3c8a
Oct 30 04:00:34.413172 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:34.413128    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" event={"ID":"e1064998-e955-4455-8402-bc240e20b88b","Type":"ContainerStarted","Data":"2ceb1eb2b3e5d65abcb3d04fec78ea43391a9a0dfc2f849b5dd5342434984bd6"}
Oct 30 04:00:34.413358 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:34.413198    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" event={"ID":"e1064998-e955-4455-8402-bc240e20b88b","Type":"ContainerStarted","Data":"0ee685cd1f7afc5c21af9984947678d9ad2fa3bab429b5611e86108d942f3c8a"}
Oct 30 04:00:39.436984 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:39.436954    4885 generic.go:356] "Generic (PLEG): container finished" podID="e1064998-e955-4455-8402-bc240e20b88b" containerID="2ceb1eb2b3e5d65abcb3d04fec78ea43391a9a0dfc2f849b5dd5342434984bd6" exitCode=0
Oct 30 04:00:39.437361 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:39.437024    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" event={"ID":"e1064998-e955-4455-8402-bc240e20b88b","Type":"ContainerDied","Data":"2ceb1eb2b3e5d65abcb3d04fec78ea43391a9a0dfc2f849b5dd5342434984bd6"}
Oct 30 04:00:40.445149 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:40.445109    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" event={"ID":"e1064998-e955-4455-8402-bc240e20b88b","Type":"ContainerStarted","Data":"69409ce07c627d5102af669a453823762228af7f2af31d874a5c2cd45669a794"}
Oct 30 04:00:40.445519 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:40.445200    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" event={"ID":"e1064998-e955-4455-8402-bc240e20b88b","Type":"ContainerStarted","Data":"17c74c561a626911928030e5e3a7727b2cf7a6267c42bca196d7dd62265693cc"}
Oct 30 04:00:40.445519 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:40.445494    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:40.445669 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:40.445651    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:40.446947 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:40.446922    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.95:8080: connect: connection refused"
Oct 30 04:00:40.498115 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:40.498043    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" podStartSLOduration=2.895538252 podStartE2EDuration="7.498023458s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.602485206s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:40.496649063 +0000 UTC m=+3946.006818089" watchObservedRunningTime="2026-10-30 04:00:40.498023458 +0000 UTC m=+3946.008192481"
Oct 30 04:00:41.449677 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:41.449634    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.95:8080: connect: connection refused"
Oct 30 04:00:46.454308 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:46.454278    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:00:46.454887 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:46.454853    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.95:8080: connect: connection refused"
Oct 30 04:00:55.921291 ip-10-0-140-132 kubenswrapper[4885]: E1030 04:00:55.921258    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 04:00:56.454803 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:00:56.454765    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.95:8080: connect: connection refused"
Oct 30 04:01:06.455235 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:06.455197    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.95:8080: connect: connection refused"
Oct 30 04:01:16.455715 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:16.455678    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.95:8080: connect: connection refused"
Oct 30 04:01:25.922105 ip-10-0-140-132 kubenswrapper[4885]: E1030 04:01:25.922067    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 04:01:26.456246 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:26.456220    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:01:34.753990 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:34.753955    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"]
Oct 30 04:01:34.754507 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:34.754176    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kserve-container" containerID="cri-o://17c74c561a626911928030e5e3a7727b2cf7a6267c42bca196d7dd62265693cc" gracePeriod=30
Oct 30 04:01:34.754507 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:34.754232    4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kube-rbac-proxy" containerID="cri-o://69409ce07c627d5102af669a453823762228af7f2af31d874a5c2cd45669a794" gracePeriod=30
Oct 30 04:01:35.677535 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:35.677503    4885 generic.go:356] "Generic (PLEG): container finished" podID="e1064998-e955-4455-8402-bc240e20b88b" containerID="69409ce07c627d5102af669a453823762228af7f2af31d874a5c2cd45669a794" exitCode=0
Oct 30 04:01:35.677728 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:35.677571    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" event={"ID":"e1064998-e955-4455-8402-bc240e20b88b","Type":"ContainerDied","Data":"69409ce07c627d5102af669a453823762228af7f2af31d874a5c2cd45669a794"}
Oct 30 04:01:36.450561 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:36.450507    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.95:8643/healthz\": dial tcp 10.132.0.95:8643: connect: connection refused"
Oct 30 04:01:36.454819 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:36.454791    4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.95:8080: connect: connection refused"
Oct 30 04:01:37.687987 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.687958    4885 generic.go:356] "Generic (PLEG): container finished" podID="e1064998-e955-4455-8402-bc240e20b88b" containerID="17c74c561a626911928030e5e3a7727b2cf7a6267c42bca196d7dd62265693cc" exitCode=0
Oct 30 04:01:37.688331 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.688045    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" event={"ID":"e1064998-e955-4455-8402-bc240e20b88b","Type":"ContainerDied","Data":"17c74c561a626911928030e5e3a7727b2cf7a6267c42bca196d7dd62265693cc"}
Oct 30 04:01:37.797749 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.797726    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:01:37.825867 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.825842    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1064998-e955-4455-8402-bc240e20b88b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1064998-e955-4455-8402-bc240e20b88b-proxy-tls\") pod \"e1064998-e955-4455-8402-bc240e20b88b\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") "
Oct 30 04:01:37.826030 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.825881    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-cabundle-cert\") pod \"e1064998-e955-4455-8402-bc240e20b88b\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") "
Oct 30 04:01:37.826030 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.825925    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1064998-e955-4455-8402-bc240e20b88b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1064998-e955-4455-8402-bc240e20b88b-kserve-provision-location\") pod \"e1064998-e955-4455-8402-bc240e20b88b\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") "
Oct 30 04:01:37.826030 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.825974    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-kube-rbac-proxy-sar-config\") pod \"e1064998-e955-4455-8402-bc240e20b88b\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") "
Oct 30 04:01:37.826030 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.825997    4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1064998-e955-4455-8402-bc240e20b88b-kube-api-access-k8m6d\" (UniqueName: \"kubernetes.io/projected/e1064998-e955-4455-8402-bc240e20b88b-kube-api-access-k8m6d\") pod \"e1064998-e955-4455-8402-bc240e20b88b\" (UID: \"e1064998-e955-4455-8402-bc240e20b88b\") "
Oct 30 04:01:37.826342 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.826319    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-cabundle-cert" pod "e1064998-e955-4455-8402-bc240e20b88b" (UID: "e1064998-e955-4455-8402-bc240e20b88b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:37.826414 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.826319    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-kube-rbac-proxy-sar-config" pod "e1064998-e955-4455-8402-bc240e20b88b" (UID: "e1064998-e955-4455-8402-bc240e20b88b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:37.828135 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.828111    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1064998-e955-4455-8402-bc240e20b88b-proxy-tls" pod "e1064998-e955-4455-8402-bc240e20b88b" (UID: "e1064998-e955-4455-8402-bc240e20b88b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:01:37.828232 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.828198    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1064998-e955-4455-8402-bc240e20b88b-kube-api-access-k8m6d" pod "e1064998-e955-4455-8402-bc240e20b88b" (UID: "e1064998-e955-4455-8402-bc240e20b88b"). InnerVolumeSpecName "kube-api-access-k8m6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:01:37.835396 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.835372    4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1064998-e955-4455-8402-bc240e20b88b-kserve-provision-location" pod "e1064998-e955-4455-8402-bc240e20b88b" (UID: "e1064998-e955-4455-8402-bc240e20b88b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:01:37.927250 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.927169    4885 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-cabundle-cert\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 04:01:37.927250 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.927199    4885 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1064998-e955-4455-8402-bc240e20b88b-kserve-provision-location\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 04:01:37.927250 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.927211    4885 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1064998-e955-4455-8402-bc240e20b88b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 04:01:37.927250 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.927219    4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8m6d\" (UniqueName: \"kubernetes.io/projected/e1064998-e955-4455-8402-bc240e20b88b-kube-api-access-k8m6d\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 04:01:37.927250 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:37.927228    4885 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1064998-e955-4455-8402-bc240e20b88b-proxy-tls\") on node \"ip-10-0-140-132.ec2.internal\" DevicePath \"\""
Oct 30 04:01:38.694974 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:38.694934    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9" event={"ID":"e1064998-e955-4455-8402-bc240e20b88b","Type":"ContainerDied","Data":"0ee685cd1f7afc5c21af9984947678d9ad2fa3bab429b5611e86108d942f3c8a"}
Oct 30 04:01:38.695397 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:38.694995    4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"
Oct 30 04:01:38.695397 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:38.695012    4885 scope.go:122] "RemoveContainer" containerID="69409ce07c627d5102af669a453823762228af7f2af31d874a5c2cd45669a794"
Oct 30 04:01:38.703374 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:38.703355    4885 scope.go:122] "RemoveContainer" containerID="17c74c561a626911928030e5e3a7727b2cf7a6267c42bca196d7dd62265693cc"
Oct 30 04:01:38.711555 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:38.711535    4885 scope.go:122] "RemoveContainer" containerID="2ceb1eb2b3e5d65abcb3d04fec78ea43391a9a0dfc2f849b5dd5342434984bd6"
Oct 30 04:01:38.750847 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:38.750820    4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"]
Oct 30 04:01:38.754713 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:38.754693    4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-paddle-runtime-predictor-5488b85b79-6ljz9"]
Oct 30 04:01:40.623349 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:01:40.623314    4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1064998-e955-4455-8402-bc240e20b88b" path="/var/lib/kubelet/pods/e1064998-e955-4455-8402-bc240e20b88b/volumes"
Oct 30 04:01:55.920909 ip-10-0-140-132 kubenswrapper[4885]: E1030 04:01:55.920871    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 04:02:25.922737 ip-10-0-140-132 kubenswrapper[4885]: E1030 04:02:25.922700    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 04:02:55.376911 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:02:55.376878    4885 scope.go:122] "RemoveContainer" containerID="7c218350eec44a37854e7225c7b4c107375efc8f343e1e1ff2b69eac7559e117"
Oct 30 04:02:55.385720 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:02:55.385694    4885 scope.go:122] "RemoveContainer" containerID="08020f4c35d79aceb97e24c0d9bb049fa987bc6903dff520a2a5a1d1ccd8d3fb"
Oct 30 04:02:55.393704 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:02:55.393680    4885 scope.go:122] "RemoveContainer" containerID="1cc0a9355421fdd7c17ea3afb7e07705a54bba5aa50dc4ee8c960247583a9e91"
Oct 30 04:02:55.921668 ip-10-0-140-132 kubenswrapper[4885]: E1030 04:02:55.921602    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 04:03:25.921763 ip-10-0-140-132 kubenswrapper[4885]: E1030 04:03:25.921680    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 04:03:55.921236 ip-10-0-140-132 kubenswrapper[4885]: E1030 04:03:55.921194    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 04:04:11.434339 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.434306    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s4bss/must-gather-j99cz"]
Oct 30 04:04:11.434850 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.434603    4885 cpu_manager.go:414] "containerMap: removing container" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kserve-container"
Oct 30 04:04:11.434850 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.434628    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kserve-container"
Oct 30 04:04:11.434850 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.434654    4885 cpu_manager.go:414] "containerMap: removing container" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="storage-initializer"
Oct 30 04:04:11.434850 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.434660    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="storage-initializer"
Oct 30 04:04:11.434850 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.434667    4885 cpu_manager.go:414] "containerMap: removing container" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kube-rbac-proxy"
Oct 30 04:04:11.434850 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.434671    4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kube-rbac-proxy"
Oct 30 04:04:11.434850 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.434730    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kserve-container"
Oct 30 04:04:11.434850 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.434740    4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1064998-e955-4455-8402-bc240e20b88b" containerName="kube-rbac-proxy"
Oct 30 04:04:11.437671 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.437649    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s4bss/must-gather-j99cz"
Oct 30 04:04:11.450287 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.450263    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s4bss\"/\"openshift-service-ca.crt\""
Oct 30 04:04:11.460412 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.460388    4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s4bss\"/\"kube-root-ca.crt\""
Oct 30 04:04:11.460571 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.460469    4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s4bss\"/\"default-dockercfg-gs6td\""
Oct 30 04:04:11.461782 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.461760    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s4bss/must-gather-j99cz"]
Oct 30 04:04:11.527857 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.527820    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db123ddf-ecaa-42e0-8ca8-8bd290affb39-must-gather-output\") pod \"must-gather-j99cz\" (UID: \"db123ddf-ecaa-42e0-8ca8-8bd290affb39\") " pod="openshift-must-gather-s4bss/must-gather-j99cz"
Oct 30 04:04:11.528030 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.527876    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcv7v\" (UniqueName: \"kubernetes.io/projected/db123ddf-ecaa-42e0-8ca8-8bd290affb39-kube-api-access-bcv7v\") pod \"must-gather-j99cz\" (UID: \"db123ddf-ecaa-42e0-8ca8-8bd290affb39\") " pod="openshift-must-gather-s4bss/must-gather-j99cz"
Oct 30 04:04:11.629105 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.629068    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db123ddf-ecaa-42e0-8ca8-8bd290affb39-must-gather-output\") pod \"must-gather-j99cz\" (UID: \"db123ddf-ecaa-42e0-8ca8-8bd290affb39\") " pod="openshift-must-gather-s4bss/must-gather-j99cz"
Oct 30 04:04:11.629270 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.629116    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcv7v\" (UniqueName: \"kubernetes.io/projected/db123ddf-ecaa-42e0-8ca8-8bd290affb39-kube-api-access-bcv7v\") pod \"must-gather-j99cz\" (UID: \"db123ddf-ecaa-42e0-8ca8-8bd290affb39\") " pod="openshift-must-gather-s4bss/must-gather-j99cz"
Oct 30 04:04:11.629421 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.629402    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db123ddf-ecaa-42e0-8ca8-8bd290affb39-must-gather-output\") pod \"must-gather-j99cz\" (UID: \"db123ddf-ecaa-42e0-8ca8-8bd290affb39\") " pod="openshift-must-gather-s4bss/must-gather-j99cz"
Oct 30 04:04:11.650775 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.650745    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcv7v\" (UniqueName: \"kubernetes.io/projected/db123ddf-ecaa-42e0-8ca8-8bd290affb39-kube-api-access-bcv7v\") pod \"must-gather-j99cz\" (UID: \"db123ddf-ecaa-42e0-8ca8-8bd290affb39\") " pod="openshift-must-gather-s4bss/must-gather-j99cz"
Oct 30 04:04:11.765668 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:11.765599    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s4bss/must-gather-j99cz"
Oct 30 04:04:12.058480 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:12.058399    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s4bss/must-gather-j99cz"]
Oct 30 04:04:12.061736 ip-10-0-140-132 kubenswrapper[4885]: W1030 04:04:12.061712    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb123ddf_ecaa_42e0_8ca8_8bd290affb39.slice/crio-be47cae71b5ea1642e69eb7166c0e53d5c3995a62d6c21be911ffdb98c389128 WatchSource:0}: Error finding container be47cae71b5ea1642e69eb7166c0e53d5c3995a62d6c21be911ffdb98c389128: Status 404 returned error can't find the container with id be47cae71b5ea1642e69eb7166c0e53d5c3995a62d6c21be911ffdb98c389128
Oct 30 04:04:12.063980 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:12.063961    4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:04:12.323734 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:12.323655    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4bss/must-gather-j99cz" event={"ID":"db123ddf-ecaa-42e0-8ca8-8bd290affb39","Type":"ContainerStarted","Data":"be47cae71b5ea1642e69eb7166c0e53d5c3995a62d6c21be911ffdb98c389128"}
Oct 30 04:04:13.329683 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:13.329644    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4bss/must-gather-j99cz" event={"ID":"db123ddf-ecaa-42e0-8ca8-8bd290affb39","Type":"ContainerStarted","Data":"0b952e115969c03efdcf5134ba19be164b583f517fbe8f0a6e3fd0548edcafc3"}
Oct 30 04:04:13.330070 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:13.329693    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4bss/must-gather-j99cz" event={"ID":"db123ddf-ecaa-42e0-8ca8-8bd290affb39","Type":"ContainerStarted","Data":"615c549c4928a5ec132dd6c9d72e89fc41ecf84919a0eb4373b57d85ab9ec2d5"}
Oct 30 04:04:13.371709 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:13.371599    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-s4bss/must-gather-j99cz" podStartSLOduration=1.559845287 podStartE2EDuration="2.371583633s" totalImagesPullingTime="811.738346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:13.369886483 +0000 UTC m=+4158.880055509" watchObservedRunningTime="2026-10-30 04:04:13.371583633 +0000 UTC m=+4158.881752658"
Oct 30 04:04:16.054522 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:16.054490    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ffnvh_cd7651fe-9b08-41ab-b4e4-5df2fbee97ee/global-pull-secret-syncer/0.log"
Oct 30 04:04:16.468939 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:16.468917    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6qkm5_4bfba258-be89-4d08-8691-1e7ccd648533/connection-checker/0.log"
Oct 30 04:04:16.590945 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:16.590905    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pkft5_6f24e8ae-7115-48d9-b285-738049028e7b/connection-checker/0.log"
Oct 30 04:04:16.710061 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:16.710034    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zvmbf_def369e4-1dd4-49b8-8ee7-b6b21168a589/connection-checker/0.log"
Oct 30 04:04:17.123024 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:17.122997    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x5wjw_e2811f6f-7a04-455e-953c-18c1018672d8/konnectivity-agent/0.log"
Oct 30 04:04:17.489902 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:17.489868    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-132.ec2.internal_7470c725c1ecc41c5e63c668787a572e/haproxy/0.log"
Oct 30 04:04:23.248908 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:23.248877    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-c2fsj_ded308e2-be12-49e1-9ddc-153bedb23ce1/cluster-monitoring-operator/0.log"
Oct 30 04:04:24.294741 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:24.294696    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpgj2_ebd76df4-8eba-49a7-b254-db08ffb63a04/node-exporter/0.log"
Oct 30 04:04:24.408733 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:24.408706    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpgj2_ebd76df4-8eba-49a7-b254-db08ffb63a04/kube-rbac-proxy/0.log"
Oct 30 04:04:24.523005 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:24.522978    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpgj2_ebd76df4-8eba-49a7-b254-db08ffb63a04/init-textfile/0.log"
Oct 30 04:04:25.429098 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.429068    4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"]
Oct 30 04:04:25.434388 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.434366    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.453659 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.453628    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"]
Oct 30 04:04:25.518330 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.518297    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eeee7ada-6f21-4f54-b247-865dcdcba148-podres\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.518526 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.518360    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh7jj\" (UniqueName: \"kubernetes.io/projected/eeee7ada-6f21-4f54-b247-865dcdcba148-kube-api-access-kh7jj\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.518526 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.518438    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eeee7ada-6f21-4f54-b247-865dcdcba148-proc\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.518526 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.518465    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eeee7ada-6f21-4f54-b247-865dcdcba148-lib-modules\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.518526 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.518487    4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeee7ada-6f21-4f54-b247-865dcdcba148-sys\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.619284 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.619229    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eeee7ada-6f21-4f54-b247-865dcdcba148-podres\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.619466 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.619323    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh7jj\" (UniqueName: \"kubernetes.io/projected/eeee7ada-6f21-4f54-b247-865dcdcba148-kube-api-access-kh7jj\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.619466 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.619343    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eeee7ada-6f21-4f54-b247-865dcdcba148-proc\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.619466 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.619357    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eeee7ada-6f21-4f54-b247-865dcdcba148-lib-modules\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.619466 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.619384    4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeee7ada-6f21-4f54-b247-865dcdcba148-sys\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.619466 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.619390    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eeee7ada-6f21-4f54-b247-865dcdcba148-podres\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.619748 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.619489    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eeee7ada-6f21-4f54-b247-865dcdcba148-proc\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.619748 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.619507    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeee7ada-6f21-4f54-b247-865dcdcba148-sys\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.619748 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.619511    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eeee7ada-6f21-4f54-b247-865dcdcba148-lib-modules\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.641883 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.641849    4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh7jj\" (UniqueName: \"kubernetes.io/projected/eeee7ada-6f21-4f54-b247-865dcdcba148-kube-api-access-kh7jj\") pod \"perf-node-gather-daemonset-dmng7\" (UID: \"eeee7ada-6f21-4f54-b247-865dcdcba148\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.747800 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:25.747768    4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:25.923841 ip-10-0-140-132 kubenswrapper[4885]: E1030 04:04:25.923810    4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4bvl" containerName="extractor"
Oct 30 04:04:26.059248 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:26.059159    4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"]
Oct 30 04:04:26.063602 ip-10-0-140-132 kubenswrapper[4885]: W1030 04:04:26.063567    4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeeee7ada_6f21_4f54_b247_865dcdcba148.slice/crio-c9ef904b56d03b89033847c172094e014528c754da7882b1118d66edd6dab413 WatchSource:0}: Error finding container c9ef904b56d03b89033847c172094e014528c754da7882b1118d66edd6dab413: Status 404 returned error can't find the container with id c9ef904b56d03b89033847c172094e014528c754da7882b1118d66edd6dab413
Oct 30 04:04:26.416648 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:26.416543    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7" event={"ID":"eeee7ada-6f21-4f54-b247-865dcdcba148","Type":"ContainerStarted","Data":"93764ae7ba5ecba6a5726bbfc03bad01c769e888c99b9cf836955a826290a6b2"}
Oct 30 04:04:26.416648 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:26.416635    4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7" event={"ID":"eeee7ada-6f21-4f54-b247-865dcdcba148","Type":"ContainerStarted","Data":"c9ef904b56d03b89033847c172094e014528c754da7882b1118d66edd6dab413"}
Oct 30 04:04:26.417337 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:26.417315    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:26.455064 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:26.455015    4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7" podStartSLOduration=1.455000232 podStartE2EDuration="1.455000232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:26.453559663 +0000 UTC m=+4171.963728691" watchObservedRunningTime="2026-10-30 04:04:26.455000232 +0000 UTC m=+4171.965169257"
Oct 30 04:04:28.751336 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:28.751304    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-5wtlc_c59c81e5-84cf-4660-9dd3-f3d8999812ec/networking-console-plugin/0.log"
Oct 30 04:04:29.662795 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:29.662761    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/1.log"
Oct 30 04:04:29.666606 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:29.666585    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5n6xt_68b4da6f-8901-46e0-a9c8-fe3d1bea91a7/console-operator/2.log"
Oct 30 04:04:31.576128 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:31.576102    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-7fbf2_a09a8421-0c4e-4343-90ac-a3d3b7f9ba36/volume-data-source-validator/0.log"
Oct 30 04:04:33.437983 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:33.437955    4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-dmng7"
Oct 30 04:04:33.868960 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:33.868930    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qggb8_d921d44f-5ee7-449a-87da-2ebbb52af1aa/dns/0.log"
Oct 30 04:04:33.982219 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:33.982196    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qggb8_d921d44f-5ee7-449a-87da-2ebbb52af1aa/kube-rbac-proxy/0.log"
Oct 30 04:04:34.211976 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:34.211949    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k6c24_eef4e0a8-2d45-4d38-b01d-806fdb0e4ff5/dns-node-resolver/0.log"
Oct 30 04:04:35.636544 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:35.636504    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-stg5z_eb32bcb9-2633-49f9-9e36-b378cf3a7b09/node-ca/0.log"
Oct 30 04:04:37.470088 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:37.470055    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57bd894fd7-n6trx_768098d2-ba3f-41b8-8263-b74826c98c73/router/0.log"
Oct 30 04:04:38.368378 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:38.368347    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2hw2r_5c66fc5b-5ff8-4e2b-9192-4c6b38e5e1c0/serve-healthcheck-canary/0.log"
Oct 30 04:04:39.467070 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:39.467030    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-7zx8h_2e4c1ab5-989e-4fe6-a028-85579a18a996/insights-operator/0.log"
Oct 30 04:04:39.468183 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:39.468167    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-7zx8h_2e4c1ab5-989e-4fe6-a028-85579a18a996/insights-operator/1.log"
Oct 30 04:04:39.577578 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:39.577529    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b4bvl_5255451b-5afa-4272-b134-38226bfec7d0/kube-rbac-proxy/0.log"
Oct 30 04:04:39.695434 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:39.695402    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b4bvl_5255451b-5afa-4272-b134-38226bfec7d0/exporter/0.log"
Oct 30 04:04:39.809478 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:39.809383    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b4bvl_5255451b-5afa-4272-b134-38226bfec7d0/extractor/0.log"
Oct 30 04:04:40.617545 ip-10-0-140-132 kubenswrapper[4885]: I1030 04:04:40.617512    4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-8ld9l-8zq7p_6d93c572-ccd9-4cd0-a604-72b6e33aae57/insights-gathering/0.log"