Sep 16 18:38:14.489163 ip-10-0-129-50 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:38:14.489173 ip-10-0-129-50 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:38:14.489180 ip-10-0-129-50 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:38:14.489324 ip-10-0-129-50 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:38:24.508944 ip-10-0-129-50 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:38:24.508967 ip-10-0-129-50 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5e76cf27000a4661a3c5de697da9798a --
Sep 16 18:40:04.700201 ip-10-0-129-50 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:40:05.076384 ip-10-0-129-50 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:05.076384 ip-10-0-129-50 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:40:05.076384 ip-10-0-129-50 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:05.076384 ip-10-0-129-50 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:40:05.076384 ip-10-0-129-50 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:05.077469 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.077389    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:40:05.083697 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083682    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:05.083697 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083697    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083701    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083705    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083708    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083710    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083713    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083716    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083719    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083721    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083724    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083726    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083729    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083731    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083734    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083737    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083739    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083742    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083744    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083747    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083755    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:05.083763 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083758    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083761    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083763    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083765    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083769    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083771    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083774    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083777    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083779    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083782    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083784    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083787    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083789    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083793    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083797    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083800    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083803    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083806    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083808    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:05.084243 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083811    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083813    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083815    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083819    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083823    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083825    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083828    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083831    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083834    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083836    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083839    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083842    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083845    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083848    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083850    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083855    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083857    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083860    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083862    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:05.084699 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083865    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083867    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083870    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083872    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083874    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083877    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083880    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083882    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083885    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083887    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083890    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083892    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083894    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083897    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083912    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083916    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083919    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083921    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083925    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:05.085178 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083928    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083931    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083934    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083936    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083939    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083941    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083944    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083947    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083950    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083954    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083956    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083959    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083962    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083964    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083966    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083969    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083972    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083974    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083977    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083979    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:05.085625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.083982    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084663    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084670    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084673    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084676    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084679    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084682    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084685    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084687    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084690    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084693    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084696    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084699    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084702    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084705    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084708    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084710    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084713    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084715    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084718    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:05.086110 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084721    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084725    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084727    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084731    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084735    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084737    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084741    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084744    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084747    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084750    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084753    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084757    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084760    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084762    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084765    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084768    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084771    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084773    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084776    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:05.086630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084778    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084781    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084783    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084786    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084789    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084791    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084795    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084797    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084800    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084802    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084805    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084807    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084810    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084813    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084816    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084820    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084823    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084826    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084829    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:05.087120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084832    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084834    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084837    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084840    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084842    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084845    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084848    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084850    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084853    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084855    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084858    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084860    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084862    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084865    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084867    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084870    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084872    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084875    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084878    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084880    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:05.087581 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084883    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084885    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084889    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084891    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084893    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084896    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084899    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084917    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084920    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084924    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084926    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084929    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084932    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084934    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084937    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084939    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084942    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084944    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084947    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084949    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:05.088088 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084952    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.084954    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085381    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085392    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085398    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085402    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085405    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085408    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085411    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085414    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085416    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085419    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085422    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085424    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085428    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085432    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085436    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085439    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085442    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085445    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:05.088567 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085447    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085450    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085453    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085456    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085458    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085461    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085464    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085466    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085469    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085471    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085473    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085476    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085479    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085481    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085484    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085487    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085490    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085492    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085495    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085498    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:05.089065 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085500    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085503    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085505    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085507    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085510    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085513    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085515    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085518    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085520    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085523    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085525    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085528    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085530    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085532    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085535    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085537    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085540    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085542    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085545    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085547    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:05.089557 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085553    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085555    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085558    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085561    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085564    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085568    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085572    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085575    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085578    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085581    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085583    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085586    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085589    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085591    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085594    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085596    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085599    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085601    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085604    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:05.090061 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085606    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085609    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085612    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085614    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085617    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085619    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085621    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085624    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085627    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085630    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085632    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085635    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085637    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085641    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085644    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085646    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085649    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085652    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085654    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:05.090513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085657    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085660    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.085662    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085729    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085771    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085778    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085782    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085786    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085789    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085793    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085797    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085800    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085803    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085806    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085809    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085812    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085816    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085818    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085821    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085824    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085826    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085831    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085833    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:40:05.091017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085836    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085839    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085842    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085850    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085853    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085856    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085859    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085862    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085864    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085867    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085871    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085874    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085878    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085881    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085884    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085886    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085889    2582 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085892    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085896    2582 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085900    2582 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085917    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085920    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085923    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085926    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:40:05.091588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085929    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085932    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085935    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085938    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085941    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085943    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085946    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085949    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085951    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085954    2582 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085958    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085962    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085965    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085968    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085970    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085973    2582 flags.go:64] FLAG: --help="false"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085976    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085979    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085982    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085985    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085989    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085993    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085995    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:40:05.092187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.085998    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086001    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086004    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086007    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086010    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086012    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086015    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086018    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086021    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086024    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086026    2582 flags.go:64] FLAG: --lock-file=""
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086030    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086032    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086035    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086040    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086043    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086046    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086048    2582 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086051    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086054    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086058    2582 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086061    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086065    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086068    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086072    2582 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:40:05.092821 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086075    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086078    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086081    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086084    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086087    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086090    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086093    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086099    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086102    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086105    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086108    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086110    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086115    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086118    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086121    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086124    2582 flags.go:64] FLAG: --port="10250"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086127    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086130    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04a995def5d46261b"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086133    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086137    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086140    2582 flags.go:64] FLAG: --register-node="true"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086142    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086146    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086149    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:40:05.093723 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086152    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086154    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086158    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086161    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086164    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086167    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086170    2582 flags.go:64] FLAG: --runonce="false"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086173    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086176    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086178    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086181    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086184    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086187    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086190    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086193    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086195    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086198    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086201    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086203    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086206    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086209    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086212    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086217    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086219    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086222    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:40:05.094381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086227    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086229    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086232    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086235    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086238    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086241    2582 flags.go:64] FLAG: --v="2"
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086246    2582 flags.go:64] FLAG: --version="false"
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086250    2582 flags.go:64] FLAG: --vmodule=""
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086253    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086256    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086367    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086371    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086374    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086377    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086380    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086384    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086388    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086391    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086394    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086396    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086399    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086402    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:05.095069 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086405    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086407    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086410    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086412    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086415    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086417    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086420    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086422    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086425    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086427    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086430    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086432    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086435    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086437    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086441    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086443    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086447    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086450    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086452    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086455    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:05.095600 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086457    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086461    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086465    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086468    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086471    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086474    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086476    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086479    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086481    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086484    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086486    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086489    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086491    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086493    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086496    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086498    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086501    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086503    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086506    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:05.096115 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086508    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086510    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086513    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086515    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086518    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086520    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086522    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086525    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086528    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086532    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086535    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086537    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086540    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086542    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086545    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086547    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086550    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086552    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086555    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086557    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:05.096582 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086560    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086562    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086566    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086569    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086572    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086575    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086578    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086580    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086583    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086585    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086588    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086600    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086602    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086605    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086607    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086610    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086614    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086616    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086619    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:05.097171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086621    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086624    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086628    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086631    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086634    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086636    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086639    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086642    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.086644    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.086649    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.095444    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.095460    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.096974    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.096982    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:40:05.097641 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097036    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097041    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097044    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097047    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097050    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097053    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097057    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097060    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097062    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097065    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097068    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097071    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097073    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097076    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097078    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097081    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097084    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097086    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097089    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097091    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:05.097999 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097094    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097096    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097099    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097101    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097104    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097107    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097109    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097113    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097118    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097122    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097125    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097128    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097131    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097134    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097136    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097139    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097142    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097153    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097157    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:05.098522 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097160    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097163    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097166    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097169    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097171    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097174    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097177    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097179    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097182    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097185    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097188    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097191    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097193    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097196    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097198    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097200    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097203    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097206    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097209    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097211    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:05.099008 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097213    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097217    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097220    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097223    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097225    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097228    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097230    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097234    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097237    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097240    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097243    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097246    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097248    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097251    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097253    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097255    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097258    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097260    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097263    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:05.099513 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097265    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097269    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097271    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097273    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097276    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097278    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097281    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097283    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097286    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097288    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097291    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097293    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097296    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097298    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097301    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097304    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097307    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097310    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097312    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097315    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:05.099988 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097317    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.097322    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097433    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097437    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097441    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097444    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097446    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097449    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097452    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097454    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097457    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097459    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097462    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097464    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:05.100474 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097467    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097470    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097472    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097474    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097477    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097479    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097482    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097485    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097487    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097489    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097492    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097494    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097497    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097499    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097502    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097504    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097507    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097510    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097512    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097515    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:05.100828 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097517    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097520    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097522    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097526    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097529    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097532    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097534    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097537    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097539    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097542    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097544    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097547    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097549    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097552    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097554    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097557    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097559    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097561    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097564    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097566    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:05.101418 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097569    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097571    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097573    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097576    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097578    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097581    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097584    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097586    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097589    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097591    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097594    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097596    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097598    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097601    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097604    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097606    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097609    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097611    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097614    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097616    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:05.101991 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097619    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097621    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097623    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097626    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097628    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097631    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097634    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097636    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097639    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097642    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097644    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097647    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097649    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097652    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097655    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097658    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097662    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097664    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097667    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:05.102482 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097670    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097673    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097676    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097678    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097681    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097684    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097687    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:05.097690    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.097694    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.097782    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.100361    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.101084    2582 server.go:1019] "Starting client certificate rotation"
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.101195    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:05.103015 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.101812    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:05.103418 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.103406    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:05.104518 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.104498    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:05.116517 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.116503    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:40:05.120303 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.120293    2582 log.go:25] "Validated CRI v1 image API"
Sep 16 18:40:05.121194 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.121176    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:40:05.123591 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.123573    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bcf1d7f7-2da7-47d7-9bc9-ca6bcdca7d13:/dev/nvme0n1p4 e1a9fa46-b390-4c85-a18a-2aed8c4017c7:/dev/nvme0n1p3]
Sep 16 18:40:05.123654 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.123590    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:40:05.128354 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.128256    2582 manager.go:216] Machine: {Timestamp:2026-09-16 18:40:05.126994529 +0000 UTC m=+0.314370307 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093278 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2277481eefa3f52862fe5f813f2da0 SystemUUID:ec227748-1eef-a3f5-2862-fe5f813f2da0 BootID:5e76cf27-000a-4661-a3c5-de697da9798a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:2b:56:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:2b:56:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:1f:fd:ae:13:d9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:40:05.128354 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.128350    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:40:05.128463 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.128415    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:40:05.132267 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.131741    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:40:05.132612 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.132033    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-50.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:40:05.133127 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.133114    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:40:05.133177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.133130    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:40:05.133177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.133138    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:05.133679 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.133669    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:05.134421 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.134411    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:40:05.137262 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.137250    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:05.137377 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.137367    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:40:05.137408 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.137398    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:40:05.137440 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.137436    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:40:05.137537 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.137526    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:40:05.137599 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.137542    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:40:05.137599 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.137553    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:40:05.137685 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.137634    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:05.137944 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.137923    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:05.137944 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.137945    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:05.140176 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.140152    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:40:05.141770 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.141756    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:40:05.141835 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.141786    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:40:05.143121 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.143110    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:40:05.143160 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.143127    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:40:05.143160 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.143132    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:40:05.143160 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.143137    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:40:05.143160 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.143145    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:40:05.143160 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.143150    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:40:05.143160 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.143161    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:40:05.143355 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.143166    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:40:05.143355 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.143172    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:40:05.143355 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.143177    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:40:05.143355 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.143185    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:40:05.143355 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.143189    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:40:05.144403 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.144390    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:40:05.144403 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.144408    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:40:05.148030 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.147894    2582 server.go:1268] "Started kubelet"
Sep 16 18:40:05.148030 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.147997    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:40:05.148126 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.148056    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:40:05.148126 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.147973    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:40:05.148785 ip-10-0-129-50 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:40:05.149589 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.149483    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-50.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:40:05.149714 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.149664    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:40:05.150189 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.150110    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:40:05.151851 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.151819    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:40:05.151851 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.151826    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:40:05.154185 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.154168    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:40:05.155363 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.155348    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:40:05.155454 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.155389    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:05.155515 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.155459    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:05.156023 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.156010    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:40:05.156113 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.156021    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:40:05.156113 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.156057    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:40:05.156215 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.156180    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:40:05.156215 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.156189    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:40:05.156477 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.156460    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:40:05.156477 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.156476    2582 factory.go:55] Registering systemd factory
Sep 16 18:40:05.156623 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.156486    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:40:05.156789 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.156778    2582 factory.go:153] Registering CRI-O factory
Sep 16 18:40:05.156789 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.156791    2582 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:40:05.156891 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.156803    2582 factory.go:103] Registering Raw factory
Sep 16 18:40:05.156891 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.156813    2582 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:40:05.157317 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.157297    2582 manager.go:349] Starting recovery of all containers
Sep 16 18:40:05.157530 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.156470    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-50.ec2.internal.18d5e12bca8e5ca3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-50.ec2.internal,UID:ip-10-0-129-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-50.ec2.internal,},FirstTimestamp:2026-09-16 18:40:05.147810979 +0000 UTC m=+0.335186757,LastTimestamp:2026-09-16 18:40:05.147810979 +0000 UTC m=+0.335186757,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-50.ec2.internal,}"
Sep 16 18:40:05.164399 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.164367    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:40:05.164514 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.164417    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:40:05.170741 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.170725    2582 manager.go:354] Recovery completed
Sep 16 18:40:05.171945 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.171919    2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 18:40:05.174176 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.174158    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x5n27"
Sep 16 18:40:05.174950 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.174935    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:05.176631 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.176616    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:05.176681 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.176645    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:05.176681 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.176659    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:05.177164 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.177142    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:40:05.177226 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.177166    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:40:05.177226 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.177183    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:05.177226 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.177184    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:05.177787 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.177774    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:05.177824 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.177798    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:05.177824 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.177810    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:05.178498 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.178487    2582 policy_none.go:49] "None policy: Start"
Sep 16 18:40:05.178531 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.178503    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:40:05.178531 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.178513    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:40:05.178840 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.178783    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-50.ec2.internal.18d5e12bcc4621ed  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-50.ec2.internal,UID:ip-10-0-129-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-50.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-50.ec2.internal,},FirstTimestamp:2026-09-16 18:40:05.176631789 +0000 UTC m=+0.364007567,LastTimestamp:2026-09-16 18:40:05.176631789 +0000 UTC m=+0.364007567,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-50.ec2.internal,}"
Sep 16 18:40:05.179520 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.179509    2582 policy_none.go:47] "Start"
Sep 16 18:40:05.184730 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.184714    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x5n27"
Sep 16 18:40:05.222470 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.222456    2582 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:40:05.232343 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.222493    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:40:05.232343 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.222502    2582 server.go:87] "Starting device plugin registration server"
Sep 16 18:40:05.232343 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.222741    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:40:05.232343 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.222750    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:40:05.232343 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.222934    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:40:05.232343 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.223018    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:40:05.232343 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.223029    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:40:05.232343 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.223370    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:40:05.232343 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.223409    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:05.289321 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.289297    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:40:05.290533 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.290511    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:40:05.290618 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.290566    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:40:05.290618 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.290589    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:40:05.290715 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.290628    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:40:05.293045 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.293029    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:05.323138 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.323114    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:05.323850 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.323837    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:05.323940 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.323862    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:05.323940 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.323871    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:05.323940 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.323898    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.330467 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.330418    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.330467 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.330436    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-50.ec2.internal\": node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:05.345479 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.345459    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:05.391663 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.391644    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"]
Sep 16 18:40:05.391722 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.391705    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:05.392858 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.392839    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:05.392959 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.392869    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:05.392959 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.392881    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:05.394109 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.394096    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:05.394246 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.394231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.394307 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.394260    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:05.394745 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.394731    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:05.394810 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.394750    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:05.394810 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.394756    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:05.394810 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.394767    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:05.394810 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.394768    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:05.394810 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.394777    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:05.395994 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.395976    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.396084 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.396000    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:05.396739 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.396712    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:05.396739 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.396731    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:05.396865 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.396743    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:05.421665 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.421648    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-50.ec2.internal\" not found" node="ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.426091 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.426076    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-50.ec2.internal\" not found" node="ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.445519 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.445496    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:05.458094 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.458074    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e9ace810eda97af717471d4045e5e4a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal\" (UID: \"9e9ace810eda97af717471d4045e5e4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.458161 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.458097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e9ace810eda97af717471d4045e5e4a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal\" (UID: \"9e9ace810eda97af717471d4045e5e4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.458161 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.458115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bcc40a76f3cacad9700ee6d1963bc3a0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-50.ec2.internal\" (UID: \"bcc40a76f3cacad9700ee6d1963bc3a0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.546586 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.546564    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:05.559152 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.559127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bcc40a76f3cacad9700ee6d1963bc3a0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-50.ec2.internal\" (UID: \"bcc40a76f3cacad9700ee6d1963bc3a0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.559249 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.559163    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e9ace810eda97af717471d4045e5e4a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal\" (UID: \"9e9ace810eda97af717471d4045e5e4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.559249 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.559186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e9ace810eda97af717471d4045e5e4a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal\" (UID: \"9e9ace810eda97af717471d4045e5e4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.559348 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.559248    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e9ace810eda97af717471d4045e5e4a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal\" (UID: \"9e9ace810eda97af717471d4045e5e4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.559348 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.559263    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bcc40a76f3cacad9700ee6d1963bc3a0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-50.ec2.internal\" (UID: \"bcc40a76f3cacad9700ee6d1963bc3a0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.559348 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.559254    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e9ace810eda97af717471d4045e5e4a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal\" (UID: \"9e9ace810eda97af717471d4045e5e4a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.647155 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.647094    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:05.723608 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.723586    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.729362 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.729344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:05.748033 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.748007    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:05.848543 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.848514    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:05.949040 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:05.948969    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:05.996069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:05.996043    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:06.046048 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.046023    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:06.049122 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:06.049100    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:06.101558 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.101535    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:40:06.101947 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.101655    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:06.101947 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.101662    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:06.101947 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.101656    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:06.150082 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:06.150058    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:06.188208 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.188173    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:35:06 +0000 UTC" deadline="2028-03-31 14:22:19.553640174 +0000 UTC"
Sep 16 18:40:06.188208 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.188200    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13483h42m13.365442661s"
Sep 16 18:40:06.227697 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:06.227667    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc40a76f3cacad9700ee6d1963bc3a0.slice/crio-490037398b7cbfa7797a25238feeea94a69d84d33203aaf47bb3235bcf4de6a5 WatchSource:0}: Error finding container 490037398b7cbfa7797a25238feeea94a69d84d33203aaf47bb3235bcf4de6a5: Status 404 returned error can't find the container with id 490037398b7cbfa7797a25238feeea94a69d84d33203aaf47bb3235bcf4de6a5
Sep 16 18:40:06.227919 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:06.227881    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9ace810eda97af717471d4045e5e4a.slice/crio-882b871973dbc9bed086624a616b1ad5fa6d87cdfc9bcb5ae8ea147691bf416b WatchSource:0}: Error finding container 882b871973dbc9bed086624a616b1ad5fa6d87cdfc9bcb5ae8ea147691bf416b: Status 404 returned error can't find the container with id 882b871973dbc9bed086624a616b1ad5fa6d87cdfc9bcb5ae8ea147691bf416b
Sep 16 18:40:06.232087 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.232065    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:40:06.250694 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:06.250664    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:06.294150 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.294109    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal" event={"ID":"9e9ace810eda97af717471d4045e5e4a","Type":"ContainerStarted","Data":"882b871973dbc9bed086624a616b1ad5fa6d87cdfc9bcb5ae8ea147691bf416b"}
Sep 16 18:40:06.295127 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.295108    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal" event={"ID":"bcc40a76f3cacad9700ee6d1963bc3a0","Type":"ContainerStarted","Data":"490037398b7cbfa7797a25238feeea94a69d84d33203aaf47bb3235bcf4de6a5"}
Sep 16 18:40:06.351279 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:06.351260    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-50.ec2.internal\" not found"
Sep 16 18:40:06.405271 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.405252    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:06.456135 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.455990    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:06.468712 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.468666    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:06.469650 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.469638    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal"
Sep 16 18:40:06.478613 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:06.478593    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:07.138157 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.138081    2582 apiserver.go:52] "Watching apiserver"
Sep 16 18:40:07.145388 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.145364    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:40:07.146691 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.146659    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-pkx6x","kube-system/global-pull-secret-syncer-stn2j","kube-system/konnectivity-agent-bjbtm","kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7","openshift-image-registry/node-ca-shplt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal","openshift-multus/multus-x4jf2","openshift-ovn-kubernetes/ovnkube-node-xxbd8","openshift-cluster-node-tuning-operator/tuned-m8d2k","openshift-dns/node-resolver-z5pl8","openshift-multus/multus-additional-cni-plugins-gnnjq","openshift-multus/network-metrics-daemon-2bzzg","openshift-network-diagnostics/network-check-target-p52xv"]
Sep 16 18:40:07.148259 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.148231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:07.148376 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.148351    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:07.149916 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.149899    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:07.150040 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.149969    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:07.151254 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.151228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bjbtm"
Sep 16 18:40:07.152738 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.152714    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pkx6x"
Sep 16 18:40:07.154312 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.153877    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:40:07.154312 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.153888    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:40:07.154312 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.153878    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hdgp7\""
Sep 16 18:40:07.154312 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.154299    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.155050 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.155029    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:40:07.156037 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.156017    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:07.157098 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.157072    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-shplt"
Sep 16 18:40:07.157618 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.157597    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:07.157739 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.157718    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:07.157824 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.157767    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kxdmp\""
Sep 16 18:40:07.157824 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.157803    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:07.157969 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.157872    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:07.158072 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.158054    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:40:07.158132 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.158109    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:40:07.158204 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.158186    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:40:07.158262 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.158207    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m77st\""
Sep 16 18:40:07.158984 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.158945    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.159622 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.159601    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:40:07.159748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.159705    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xxwm4\""
Sep 16 18:40:07.161522 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.161174    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:40:07.161522 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.161233    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:40:07.165370 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.162675    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:40:07.165370 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.162706    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:40:07.165370 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.162761    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:40:07.165370 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.162912    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:40:07.165370 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.163195    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.165370 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.163371    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdj2f\""
Sep 16 18:40:07.165370 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.164321    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.166681 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.166660    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:40:07.166985 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.166900    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6hlsl\""
Sep 16 18:40:07.167062 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzs9k\" (UniqueName: \"kubernetes.io/projected/47c4bc73-2eff-41ea-91fc-5acce1bcd526-kube-api-access-kzs9k\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:07.167062 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-registration-dir\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.167135 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10013ea8-6367-4299-ae77-de784d9ac6cc-konnectivity-ca\") pod \"konnectivity-agent-bjbtm\" (UID: \"10013ea8-6367-4299-ae77-de784d9ac6cc\") " pod="kube-system/konnectivity-agent-bjbtm"
Sep 16 18:40:07.167135 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d37b1617-d4af-42a9-8795-2074cbf8be05-serviceca\") pod \"node-ca-shplt\" (UID: \"d37b1617-d4af-42a9-8795-2074cbf8be05\") " pod="openshift-image-registry/node-ca-shplt"
Sep 16 18:40:07.167135 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.166909    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:40:07.167233 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/935eb2ed-1f77-4417-ad43-71cb39d08c66-kubelet-config\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:07.167233 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167160    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/935eb2ed-1f77-4417-ad43-71cb39d08c66-dbus\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:07.167233 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-device-dir\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.167233 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10013ea8-6367-4299-ae77-de784d9ac6cc-agent-certs\") pod \"konnectivity-agent-bjbtm\" (UID: \"10013ea8-6367-4299-ae77-de784d9ac6cc\") " pod="kube-system/konnectivity-agent-bjbtm"
Sep 16 18:40:07.167358 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f54367c0-5c70-4b72-9a0f-ee5182e7e088-iptables-alerter-script\") pod \"iptables-alerter-pkx6x\" (UID: \"f54367c0-5c70-4b72-9a0f-ee5182e7e088\") " pod="openshift-network-operator/iptables-alerter-pkx6x"
Sep 16 18:40:07.167358 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bh7w\" (UniqueName: \"kubernetes.io/projected/f54367c0-5c70-4b72-9a0f-ee5182e7e088-kube-api-access-8bh7w\") pod \"iptables-alerter-pkx6x\" (UID: \"f54367c0-5c70-4b72-9a0f-ee5182e7e088\") " pod="openshift-network-operator/iptables-alerter-pkx6x"
Sep 16 18:40:07.167358 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d37b1617-d4af-42a9-8795-2074cbf8be05-host\") pod \"node-ca-shplt\" (UID: \"d37b1617-d4af-42a9-8795-2074cbf8be05\") " pod="openshift-image-registry/node-ca-shplt"
Sep 16 18:40:07.167358 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpd7q\" (UniqueName: \"kubernetes.io/projected/d37b1617-d4af-42a9-8795-2074cbf8be05-kube-api-access-fpd7q\") pod \"node-ca-shplt\" (UID: \"d37b1617-d4af-42a9-8795-2074cbf8be05\") " pod="openshift-image-registry/node-ca-shplt"
Sep 16 18:40:07.167358 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-sys-fs\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.167505 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:07.167505 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-socket-dir\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.167505 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-etc-selinux\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.167505 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-285jk\" (UniqueName: \"kubernetes.io/projected/5a0a56bd-3ee3-41f4-9730-61b055862970-kube-api-access-285jk\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.167505 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6zv\" (UniqueName: \"kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv\") pod \"network-check-target-p52xv\" (UID: \"056ad46e-0656-444d-bed0-680d3562ee88\") " pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:07.167650 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:07.167650 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.167650 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f54367c0-5c70-4b72-9a0f-ee5182e7e088-host-slash\") pod \"iptables-alerter-pkx6x\" (UID: \"f54367c0-5c70-4b72-9a0f-ee5182e7e088\") " pod="openshift-network-operator/iptables-alerter-pkx6x"
Sep 16 18:40:07.167650 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167577    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:40:07.167903 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167778    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:07.167903 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167789    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:07.167903 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167841    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:40:07.168083 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.167935    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:40:07.168083 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.168021    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:40:07.168192 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.168175    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z5pl8"
Sep 16 18:40:07.168192 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.168184    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-49t4m\""
Sep 16 18:40:07.169045 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.169026    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.171077 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.171059    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:40:07.171300 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.171151    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kwhz8\""
Sep 16 18:40:07.171744 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.171723    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:40:07.172020 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.171630    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:40:07.172109 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.172072    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:40:07.172687 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.172669    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7wwdn\""
Sep 16 18:40:07.178598 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.178573    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:07.212203 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.212181    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hxnq9"
Sep 16 18:40:07.246110 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.246084    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hxnq9"
Sep 16 18:40:07.257277 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.257254    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:40:07.267755 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.267728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-sys-fs\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.267860 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.267769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-etc-selinux\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.267860 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.267804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.267860 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.267830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-lib-modules\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.267860 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.267855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-host\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.268128 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.267878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78095364-693e-4375-ae1e-cd23e8a6001d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.268128 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.267907    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-multus-cni-dir\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.268128 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.267880    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-sys-fs\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.268128 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.267942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-etc-selinux\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.268128 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.267967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-cnibin\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.268128 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd12d598-68f2-4545-a589-be29dda56485-cni-binary-copy\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.268128 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-multus-socket-dir-parent\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.268128 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-run-k8s-cni-cncf-io\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.268516 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-etc-kubernetes\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.268516 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v24g\" (UniqueName: \"kubernetes.io/projected/dd12d598-68f2-4545-a589-be29dda56485-kube-api-access-4v24g\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.268516 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzs9k\" (UniqueName: \"kubernetes.io/projected/47c4bc73-2eff-41ea-91fc-5acce1bcd526-kube-api-access-kzs9k\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:07.268516 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-var-lib-cni-bin\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.268516 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-kubernetes\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.268516 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268314    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8eeb34d-7a85-40ac-8755-be657c455ec6-hosts-file\") pod \"node-resolver-z5pl8\" (UID: \"f8eeb34d-7a85-40ac-8755-be657c455ec6\") " pod="openshift-dns/node-resolver-z5pl8"
Sep 16 18:40:07.268516 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggfc6\" (UniqueName: \"kubernetes.io/projected/78095364-693e-4375-ae1e-cd23e8a6001d-kube-api-access-ggfc6\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.268516 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-device-dir\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.268516 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f54367c0-5c70-4b72-9a0f-ee5182e7e088-iptables-alerter-script\") pod \"iptables-alerter-pkx6x\" (UID: \"f54367c0-5c70-4b72-9a0f-ee5182e7e088\") " pod="openshift-network-operator/iptables-alerter-pkx6x"
Sep 16 18:40:07.268802 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268520    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-device-dir\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.268802 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268571    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.268802 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268618    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-env-overrides\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.268802 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-socket-dir\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.268802 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-systemd-units\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.268802 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268734    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-cni-bin\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.268802 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8eeb34d-7a85-40ac-8755-be657c455ec6-tmp-dir\") pod \"node-resolver-z5pl8\" (UID: \"f8eeb34d-7a85-40ac-8755-be657c455ec6\") " pod="openshift-dns/node-resolver-z5pl8"
Sep 16 18:40:07.268802 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268779    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-socket-dir\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.269060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.269060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268852    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-run-netns\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.269060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268867    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-run-multus-certs\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.269060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268882    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-run\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.269060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10013ea8-6367-4299-ae77-de784d9ac6cc-konnectivity-ca\") pod \"konnectivity-agent-bjbtm\" (UID: \"10013ea8-6367-4299-ae77-de784d9ac6cc\") " pod="kube-system/konnectivity-agent-bjbtm"
Sep 16 18:40:07.269060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268956    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.269060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d37b1617-d4af-42a9-8795-2074cbf8be05-serviceca\") pod \"node-ca-shplt\" (UID: \"d37b1617-d4af-42a9-8795-2074cbf8be05\") " pod="openshift-image-registry/node-ca-shplt"
Sep 16 18:40:07.269060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.268984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f54367c0-5c70-4b72-9a0f-ee5182e7e088-iptables-alerter-script\") pod \"iptables-alerter-pkx6x\" (UID: \"f54367c0-5c70-4b72-9a0f-ee5182e7e088\") " pod="openshift-network-operator/iptables-alerter-pkx6x"
Sep 16 18:40:07.269060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-run-netns\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.269060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-ovnkube-config\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-ovnkube-script-lib\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-sysconfig\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269160    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78095364-693e-4375-ae1e-cd23e8a6001d-system-cni-dir\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269209    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10013ea8-6367-4299-ae77-de784d9ac6cc-agent-certs\") pod \"konnectivity-agent-bjbtm\" (UID: \"10013ea8-6367-4299-ae77-de784d9ac6cc\") " pod="kube-system/konnectivity-agent-bjbtm"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpd7q\" (UniqueName: \"kubernetes.io/projected/d37b1617-d4af-42a9-8795-2074cbf8be05-kube-api-access-fpd7q\") pod \"node-ca-shplt\" (UID: \"d37b1617-d4af-42a9-8795-2074cbf8be05\") " pod="openshift-image-registry/node-ca-shplt"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269313    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78095364-693e-4375-ae1e-cd23e8a6001d-cni-binary-copy\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-285jk\" (UniqueName: \"kubernetes.io/projected/5a0a56bd-3ee3-41f4-9730-61b055862970-kube-api-access-285jk\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269443    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5sq\" (UniqueName: \"kubernetes.io/projected/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-kube-api-access-pp5sq\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d37b1617-d4af-42a9-8795-2074cbf8be05-serviceca\") pod \"node-ca-shplt\" (UID: \"d37b1617-d4af-42a9-8795-2074cbf8be05\") " pod="openshift-image-registry/node-ca-shplt"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-var-lib-kubelet\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10013ea8-6367-4299-ae77-de784d9ac6cc-konnectivity-ca\") pod \"konnectivity-agent-bjbtm\" (UID: \"10013ea8-6367-4299-ae77-de784d9ac6cc\") " pod="kube-system/konnectivity-agent-bjbtm"
Sep 16 18:40:07.269549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-cni-netd\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.269584    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-modprobe-d\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.269671    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs podName:47c4bc73-2eff-41ea-91fc-5acce1bcd526 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:07.769646853 +0000 UTC m=+2.957022624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs") pod "network-metrics-daemon-2bzzg" (UID: "47c4bc73-2eff-41ea-91fc-5acce1bcd526") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78095364-693e-4375-ae1e-cd23e8a6001d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-registration-dir\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-var-lib-cni-multus\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-sys\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/935eb2ed-1f77-4417-ad43-71cb39d08c66-kubelet-config\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5a0a56bd-3ee3-41f4-9730-61b055862970-registration-dir\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269895    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/935eb2ed-1f77-4417-ad43-71cb39d08c66-dbus\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269910    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.269988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/935eb2ed-1f77-4417-ad43-71cb39d08c66-kubelet-config\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6zv\" (UniqueName: \"kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv\") pod \"network-check-target-p52xv\" (UID: \"056ad46e-0656-444d-bed0-680d3562ee88\") " pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-slash\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270082    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-var-lib-openvswitch\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-node-log\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.270219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-ovn-node-metrics-cert\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-sysctl-conf\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-kubelet\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270222    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-etc-openvswitch\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-run-openvswitch\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270279    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-var-lib-kubelet\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-tuned\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/935eb2ed-1f77-4417-ad43-71cb39d08c66-dbus\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78095364-693e-4375-ae1e-cd23e8a6001d-cnibin\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f54367c0-5c70-4b72-9a0f-ee5182e7e088-host-slash\") pod \"iptables-alerter-pkx6x\" (UID: \"f54367c0-5c70-4b72-9a0f-ee5182e7e088\") " pod="openshift-network-operator/iptables-alerter-pkx6x"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-multus-conf-dir\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270521    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f54367c0-5c70-4b72-9a0f-ee5182e7e088-host-slash\") pod \"iptables-alerter-pkx6x\" (UID: \"f54367c0-5c70-4b72-9a0f-ee5182e7e088\") " pod="openshift-network-operator/iptables-alerter-pkx6x"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.270550    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-run-ovn\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.270602    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret podName:935eb2ed-1f77-4417-ad43-71cb39d08c66 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:07.770585278 +0000 UTC m=+2.957961063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret") pod "global-pull-secret-syncer-stn2j" (UID: "935eb2ed-1f77-4417-ad43-71cb39d08c66") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:07.270915 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-sysctl-d\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fz4v\" (UniqueName: \"kubernetes.io/projected/f8eeb34d-7a85-40ac-8755-be657c455ec6-kube-api-access-9fz4v\") pod \"node-resolver-z5pl8\" (UID: \"f8eeb34d-7a85-40ac-8755-be657c455ec6\") " pod="openshift-dns/node-resolver-z5pl8"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78095364-693e-4375-ae1e-cd23e8a6001d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd12d598-68f2-4545-a589-be29dda56485-multus-daemon-config\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-run-systemd\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270761    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-tmp\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bh7w\" (UniqueName: \"kubernetes.io/projected/f54367c0-5c70-4b72-9a0f-ee5182e7e088-kube-api-access-8bh7w\") pod \"iptables-alerter-pkx6x\" (UID: \"f54367c0-5c70-4b72-9a0f-ee5182e7e088\") " pod="openshift-network-operator/iptables-alerter-pkx6x"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d37b1617-d4af-42a9-8795-2074cbf8be05-host\") pod \"node-ca-shplt\" (UID: \"d37b1617-d4af-42a9-8795-2074cbf8be05\") " pod="openshift-image-registry/node-ca-shplt"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270839    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-system-cni-dir\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-hostroot\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-log-socket\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-systemd\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.270965    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkp8b\" (UniqueName: \"kubernetes.io/projected/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-kube-api-access-jkp8b\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.271559 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.271025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d37b1617-d4af-42a9-8795-2074cbf8be05-host\") pod \"node-ca-shplt\" (UID: \"d37b1617-d4af-42a9-8795-2074cbf8be05\") " pod="openshift-image-registry/node-ca-shplt"
Sep 16 18:40:07.274474 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.274456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10013ea8-6367-4299-ae77-de784d9ac6cc-agent-certs\") pod \"konnectivity-agent-bjbtm\" (UID: \"10013ea8-6367-4299-ae77-de784d9ac6cc\") " pod="kube-system/konnectivity-agent-bjbtm"
Sep 16 18:40:07.278734 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.278686    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzs9k\" (UniqueName: \"kubernetes.io/projected/47c4bc73-2eff-41ea-91fc-5acce1bcd526-kube-api-access-kzs9k\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:07.284539 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.284520    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:07.284641 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.284546    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:07.284641 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.284560    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df6zv for pod openshift-network-diagnostics/network-check-target-p52xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:07.284641 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.284636    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv podName:056ad46e-0656-444d-bed0-680d3562ee88 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:07.784599444 +0000 UTC m=+2.971975218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-df6zv" (UniqueName: "kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv") pod "network-check-target-p52xv" (UID: "056ad46e-0656-444d-bed0-680d3562ee88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:07.287789 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.287765    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bh7w\" (UniqueName: \"kubernetes.io/projected/f54367c0-5c70-4b72-9a0f-ee5182e7e088-kube-api-access-8bh7w\") pod \"iptables-alerter-pkx6x\" (UID: \"f54367c0-5c70-4b72-9a0f-ee5182e7e088\") " pod="openshift-network-operator/iptables-alerter-pkx6x"
Sep 16 18:40:07.288273 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.288252    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-285jk\" (UniqueName: \"kubernetes.io/projected/5a0a56bd-3ee3-41f4-9730-61b055862970-kube-api-access-285jk\") pod \"aws-ebs-csi-driver-node-8kqb7\" (UID: \"5a0a56bd-3ee3-41f4-9730-61b055862970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.288684 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.288664    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpd7q\" (UniqueName: \"kubernetes.io/projected/d37b1617-d4af-42a9-8795-2074cbf8be05-kube-api-access-fpd7q\") pod \"node-ca-shplt\" (UID: \"d37b1617-d4af-42a9-8795-2074cbf8be05\") " pod="openshift-image-registry/node-ca-shplt"
Sep 16 18:40:07.371997 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.371968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-multus-conf-dir\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.372121 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-run-ovn\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.372121 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-sysctl-d\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.372121 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fz4v\" (UniqueName: \"kubernetes.io/projected/f8eeb34d-7a85-40ac-8755-be657c455ec6-kube-api-access-9fz4v\") pod \"node-resolver-z5pl8\" (UID: \"f8eeb34d-7a85-40ac-8755-be657c455ec6\") " pod="openshift-dns/node-resolver-z5pl8"
Sep 16 18:40:07.372121 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78095364-693e-4375-ae1e-cd23e8a6001d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.372121 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372088    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-multus-conf-dir\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.372121 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd12d598-68f2-4545-a589-be29dda56485-multus-daemon-config\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.372121 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372107    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-run-ovn\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-run-systemd\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-tmp\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372182    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-sysctl-d\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-system-cni-dir\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-hostroot\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-log-socket\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-run-systemd\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372298    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-systemd\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkp8b\" (UniqueName: \"kubernetes.io/projected/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-kube-api-access-jkp8b\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372369    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-systemd\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-system-cni-dir\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-lib-modules\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-host\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78095364-693e-4375-ae1e-cd23e8a6001d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372455    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-lib-modules\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372459    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-hostroot\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.372508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372468    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-multus-cni-dir\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-cnibin\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372521    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd12d598-68f2-4545-a589-be29dda56485-cni-binary-copy\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-multus-socket-dir-parent\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372594    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-multus-cni-dir\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-run-k8s-cni-cncf-io\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-etc-kubernetes\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372651    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-multus-socket-dir-parent\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372692    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v24g\" (UniqueName: \"kubernetes.io/projected/dd12d598-68f2-4545-a589-be29dda56485-kube-api-access-4v24g\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372713    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78095364-693e-4375-ae1e-cd23e8a6001d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372728    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-etc-kubernetes\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-var-lib-cni-bin\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372423    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-log-socket\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372695    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-run-k8s-cni-cncf-io\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-kubernetes\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8eeb34d-7a85-40ac-8755-be657c455ec6-hosts-file\") pod \"node-resolver-z5pl8\" (UID: \"f8eeb34d-7a85-40ac-8755-be657c455ec6\") " pod="openshift-dns/node-resolver-z5pl8"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372812    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-var-lib-cni-bin\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.373526 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372828    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggfc6\" (UniqueName: \"kubernetes.io/projected/78095364-693e-4375-ae1e-cd23e8a6001d-kube-api-access-ggfc6\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372731    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd12d598-68f2-4545-a589-be29dda56485-multus-daemon-config\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-kubernetes\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372911    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8eeb34d-7a85-40ac-8755-be657c455ec6-hosts-file\") pod \"node-resolver-z5pl8\" (UID: \"f8eeb34d-7a85-40ac-8755-be657c455ec6\") " pod="openshift-dns/node-resolver-z5pl8"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-env-overrides\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372956    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-systemd-units\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.372964    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-cnibin\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373035    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-systemd-units\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-cni-bin\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8eeb34d-7a85-40ac-8755-be657c455ec6-tmp-dir\") pod \"node-resolver-z5pl8\" (UID: \"f8eeb34d-7a85-40ac-8755-be657c455ec6\") " pod="openshift-dns/node-resolver-z5pl8"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373076    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd12d598-68f2-4545-a589-be29dda56485-cni-binary-copy\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-run-netns\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373134    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-cni-bin\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-run-multus-certs\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-run-netns\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-run\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.374424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-run-netns\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-ovnkube-config\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373227    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-run-multus-certs\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373276    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-run\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373293    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-ovnkube-script-lib\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373314    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-run-netns\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-sysconfig\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373335    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-host\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373346    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78095364-693e-4375-ae1e-cd23e8a6001d-system-cni-dir\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373367    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78095364-693e-4375-ae1e-cd23e8a6001d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373387    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-sysconfig\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78095364-693e-4375-ae1e-cd23e8a6001d-system-cni-dir\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78095364-693e-4375-ae1e-cd23e8a6001d-cni-binary-copy\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5sq\" (UniqueName: \"kubernetes.io/projected/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-kube-api-access-pp5sq\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373522    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-var-lib-kubelet\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-cni-netd\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-modprobe-d\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.375323 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78095364-693e-4375-ae1e-cd23e8a6001d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373632    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-var-lib-cni-multus\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-sys\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-slash\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373701    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-ovnkube-config\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-var-lib-openvswitch\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373740    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-node-log\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373753    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-var-lib-kubelet\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-modprobe-d\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-ovn-node-metrics-cert\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373811    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-cni-netd\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-sysctl-conf\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-kubelet\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-slash\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-etc-openvswitch\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373883    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-etc-openvswitch\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78095364-693e-4375-ae1e-cd23e8a6001d-cni-binary-copy\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-run-openvswitch\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.376252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373931    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-ovnkube-script-lib\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-run-openvswitch\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-sysctl-conf\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.374009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-var-lib-kubelet\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.374037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-tuned\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.374043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-sys\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.374046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-var-lib-openvswitch\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373491    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-env-overrides\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.373988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-host-kubelet\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.374085    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd12d598-68f2-4545-a589-be29dda56485-host-var-lib-cni-multus\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.374095    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8eeb34d-7a85-40ac-8755-be657c455ec6-tmp-dir\") pod \"node-resolver-z5pl8\" (UID: \"f8eeb34d-7a85-40ac-8755-be657c455ec6\") " pod="openshift-dns/node-resolver-z5pl8"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.374115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78095364-693e-4375-ae1e-cd23e8a6001d-cnibin\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.374089    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-var-lib-kubelet\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.374126    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-node-log\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.374138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78095364-693e-4375-ae1e-cd23e8a6001d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.374234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78095364-693e-4375-ae1e-cd23e8a6001d-cnibin\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.377172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.375217    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-tmp\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.378221 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.378193    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-ovn-node-metrics-cert\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.378934 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.378914    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-etc-tuned\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.384268 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.384140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v24g\" (UniqueName: \"kubernetes.io/projected/dd12d598-68f2-4545-a589-be29dda56485-kube-api-access-4v24g\") pod \"multus-x4jf2\" (UID: \"dd12d598-68f2-4545-a589-be29dda56485\") " pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.384586 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.384567    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkp8b\" (UniqueName: \"kubernetes.io/projected/54b82fdf-4c88-44a6-a6ac-f37f8de64fe8-kube-api-access-jkp8b\") pod \"tuned-m8d2k\" (UID: \"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8\") " pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.384669 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.384585    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fz4v\" (UniqueName: \"kubernetes.io/projected/f8eeb34d-7a85-40ac-8755-be657c455ec6-kube-api-access-9fz4v\") pod \"node-resolver-z5pl8\" (UID: \"f8eeb34d-7a85-40ac-8755-be657c455ec6\") " pod="openshift-dns/node-resolver-z5pl8"
Sep 16 18:40:07.385473 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.385443    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggfc6\" (UniqueName: \"kubernetes.io/projected/78095364-693e-4375-ae1e-cd23e8a6001d-kube-api-access-ggfc6\") pod \"multus-additional-cni-plugins-gnnjq\" (UID: \"78095364-693e-4375-ae1e-cd23e8a6001d\") " pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.385557 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.385520    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5sq\" (UniqueName: \"kubernetes.io/projected/6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098-kube-api-access-pp5sq\") pod \"ovnkube-node-xxbd8\" (UID: \"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.415114 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.414873    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:07.466671 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.466637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bjbtm"
Sep 16 18:40:07.478163 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.478142    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-shplt"
Sep 16 18:40:07.488483 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.488463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pkx6x"
Sep 16 18:40:07.494296 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.494270    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7"
Sep 16 18:40:07.501179 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.501156    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x4jf2"
Sep 16 18:40:07.509966 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.509947    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:07.517633 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.517615    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m8d2k"
Sep 16 18:40:07.525383 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.525366    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z5pl8"
Sep 16 18:40:07.531993 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.531976    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gnnjq"
Sep 16 18:40:07.718739 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.718666    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:07.778374 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.778341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:07.778524 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.778410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:07.778524 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.778511    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:07.778646 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.778581    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs podName:47c4bc73-2eff-41ea-91fc-5acce1bcd526 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:08.778560397 +0000 UTC m=+3.965936168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs") pod "network-metrics-daemon-2bzzg" (UID: "47c4bc73-2eff-41ea-91fc-5acce1bcd526") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:07.778646 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.778511    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:07.778762 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.778677    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret podName:935eb2ed-1f77-4417-ad43-71cb39d08c66 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:08.778663175 +0000 UTC m=+3.966038957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret") pod "global-pull-secret-syncer-stn2j" (UID: "935eb2ed-1f77-4417-ad43-71cb39d08c66") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:07.879227 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:07.879181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6zv\" (UniqueName: \"kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv\") pod \"network-check-target-p52xv\" (UID: \"056ad46e-0656-444d-bed0-680d3562ee88\") " pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:07.879351 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.879334    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:07.879397 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.879359    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:07.879397 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.879372    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df6zv for pod openshift-network-diagnostics/network-check-target-p52xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:07.879494 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:07.879433    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv podName:056ad46e-0656-444d-bed0-680d3562ee88 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:08.879414941 +0000 UTC m=+4.066790730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-df6zv" (UniqueName: "kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv") pod "network-check-target-p52xv" (UID: "056ad46e-0656-444d-bed0-680d3562ee88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:08.167696 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:08.167664    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8eeb34d_7a85_40ac_8755_be657c455ec6.slice/crio-6e2d1819fd7b9e3c6b88533048dab4cff8d429f56adf4012451f390ad9a9c1fb WatchSource:0}: Error finding container 6e2d1819fd7b9e3c6b88533048dab4cff8d429f56adf4012451f390ad9a9c1fb: Status 404 returned error can't find the container with id 6e2d1819fd7b9e3c6b88533048dab4cff8d429f56adf4012451f390ad9a9c1fb
Sep 16 18:40:08.169530 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:08.169384    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10013ea8_6367_4299_ae77_de784d9ac6cc.slice/crio-428c77ef52e96fab2b8f1e42ae490fbc4fba2c95392d4cdbc2e6d3e527693a05 WatchSource:0}: Error finding container 428c77ef52e96fab2b8f1e42ae490fbc4fba2c95392d4cdbc2e6d3e527693a05: Status 404 returned error can't find the container with id 428c77ef52e96fab2b8f1e42ae490fbc4fba2c95392d4cdbc2e6d3e527693a05
Sep 16 18:40:08.170549 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:08.170497    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0a56bd_3ee3_41f4_9730_61b055862970.slice/crio-42bceb104fefb9c716a48f08fbc38660886b611354038db920d031b9d138f76c WatchSource:0}: Error finding container 42bceb104fefb9c716a48f08fbc38660886b611354038db920d031b9d138f76c: Status 404 returned error can't find the container with id 42bceb104fefb9c716a48f08fbc38660886b611354038db920d031b9d138f76c
Sep 16 18:40:08.172854 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:08.172719    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37b1617_d4af_42a9_8795_2074cbf8be05.slice/crio-9b1c357fbc82fcc874fde757ae9164497938878ddfe84b205fbf29169fb98378 WatchSource:0}: Error finding container 9b1c357fbc82fcc874fde757ae9164497938878ddfe84b205fbf29169fb98378: Status 404 returned error can't find the container with id 9b1c357fbc82fcc874fde757ae9164497938878ddfe84b205fbf29169fb98378
Sep 16 18:40:08.177260 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:08.177003    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b82fdf_4c88_44a6_a6ac_f37f8de64fe8.slice/crio-716bed0364bc760d5b7c3c49ee8fc79ac3fbf3149eed104b8e3779d4b6885861 WatchSource:0}: Error finding container 716bed0364bc760d5b7c3c49ee8fc79ac3fbf3149eed104b8e3779d4b6885861: Status 404 returned error can't find the container with id 716bed0364bc760d5b7c3c49ee8fc79ac3fbf3149eed104b8e3779d4b6885861
Sep 16 18:40:08.178899 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:08.178878    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78095364_693e_4375_ae1e_cd23e8a6001d.slice/crio-a06c37a7f38b2ef62a78c2298b01ef094110523c66f68790786534dca5d39afc WatchSource:0}: Error finding container a06c37a7f38b2ef62a78c2298b01ef094110523c66f68790786534dca5d39afc: Status 404 returned error can't find the container with id a06c37a7f38b2ef62a78c2298b01ef094110523c66f68790786534dca5d39afc
Sep 16 18:40:08.181254 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:08.181215    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6c78ec_8ef8_4148_bff5_c7d9e7aa2098.slice/crio-626b4b94f3cc244c52430760fea86b3521c8540d92e3f57bed43cfef4ae69768 WatchSource:0}: Error finding container 626b4b94f3cc244c52430760fea86b3521c8540d92e3f57bed43cfef4ae69768: Status 404 returned error can't find the container with id 626b4b94f3cc244c52430760fea86b3521c8540d92e3f57bed43cfef4ae69768
Sep 16 18:40:08.248690 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.248648    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:08 +0000 UTC" deadline="2028-02-28 15:06:45.311200348 +0000 UTC"
Sep 16 18:40:08.248690 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.248689    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12716h26m37.062514354s"
Sep 16 18:40:08.301009 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.300980    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pkx6x" event={"ID":"f54367c0-5c70-4b72-9a0f-ee5182e7e088","Type":"ContainerStarted","Data":"7681385f5f17bcd764bf76f0bd21e2c870356941ec1324916f315aad25e33a7e"}
Sep 16 18:40:08.302834 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.302811    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m8d2k" event={"ID":"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8","Type":"ContainerStarted","Data":"716bed0364bc760d5b7c3c49ee8fc79ac3fbf3149eed104b8e3779d4b6885861"}
Sep 16 18:40:08.303902 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.303882    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-shplt" event={"ID":"d37b1617-d4af-42a9-8795-2074cbf8be05","Type":"ContainerStarted","Data":"9b1c357fbc82fcc874fde757ae9164497938878ddfe84b205fbf29169fb98378"}
Sep 16 18:40:08.306629 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.306606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7" event={"ID":"5a0a56bd-3ee3-41f4-9730-61b055862970","Type":"ContainerStarted","Data":"42bceb104fefb9c716a48f08fbc38660886b611354038db920d031b9d138f76c"}
Sep 16 18:40:08.314168 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.314144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal" event={"ID":"bcc40a76f3cacad9700ee6d1963bc3a0","Type":"ContainerStarted","Data":"16240e5d99d19670964d47838bfab2ebd71cf75168455ea8731f10e16811434d"}
Sep 16 18:40:08.317164 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.317141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x4jf2" event={"ID":"dd12d598-68f2-4545-a589-be29dda56485","Type":"ContainerStarted","Data":"13219a4d35fd7e37f193206e9c9e4041c592a146b304a5f228b3799e8a1d51d9"}
Sep 16 18:40:08.324189 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.319571    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8" event={"ID":"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098","Type":"ContainerStarted","Data":"626b4b94f3cc244c52430760fea86b3521c8540d92e3f57bed43cfef4ae69768"}
Sep 16 18:40:08.324189 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.320944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnnjq" event={"ID":"78095364-693e-4375-ae1e-cd23e8a6001d","Type":"ContainerStarted","Data":"a06c37a7f38b2ef62a78c2298b01ef094110523c66f68790786534dca5d39afc"}
Sep 16 18:40:08.324189 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.322004    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bjbtm" event={"ID":"10013ea8-6367-4299-ae77-de784d9ac6cc","Type":"ContainerStarted","Data":"428c77ef52e96fab2b8f1e42ae490fbc4fba2c95392d4cdbc2e6d3e527693a05"}
Sep 16 18:40:08.324189 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.323030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z5pl8" event={"ID":"f8eeb34d-7a85-40ac-8755-be657c455ec6","Type":"ContainerStarted","Data":"6e2d1819fd7b9e3c6b88533048dab4cff8d429f56adf4012451f390ad9a9c1fb"}
Sep 16 18:40:08.331004 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.330935    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-50.ec2.internal" podStartSLOduration=1.330920166 podStartE2EDuration="1.330920166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:08.330681059 +0000 UTC m=+3.518056850" watchObservedRunningTime="2026-09-16 18:40:08.330920166 +0000 UTC m=+3.518295957"
Sep 16 18:40:08.785353 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.785269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:08.785353 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.785347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:08.785567 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:08.785491    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:08.785567 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:08.785552    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret podName:935eb2ed-1f77-4417-ad43-71cb39d08c66 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:10.785532212 +0000 UTC m=+5.972907980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret") pod "global-pull-secret-syncer-stn2j" (UID: "935eb2ed-1f77-4417-ad43-71cb39d08c66") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:08.785689 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:08.785619    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:08.785689 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:08.785653    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs podName:47c4bc73-2eff-41ea-91fc-5acce1bcd526 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:10.785641788 +0000 UTC m=+5.973017559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs") pod "network-metrics-daemon-2bzzg" (UID: "47c4bc73-2eff-41ea-91fc-5acce1bcd526") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:08.886596 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:08.886556    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6zv\" (UniqueName: \"kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv\") pod \"network-check-target-p52xv\" (UID: \"056ad46e-0656-444d-bed0-680d3562ee88\") " pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:08.886802 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:08.886776    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:08.886887 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:08.886834    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:08.886887 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:08.886849    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df6zv for pod openshift-network-diagnostics/network-check-target-p52xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:08.887027 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:08.886908    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv podName:056ad46e-0656-444d-bed0-680d3562ee88 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:10.886889458 +0000 UTC m=+6.074265231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-df6zv" (UniqueName: "kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv") pod "network-check-target-p52xv" (UID: "056ad46e-0656-444d-bed0-680d3562ee88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:09.249676 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:09.249632    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:08 +0000 UTC" deadline="2028-06-22 05:06:56.735252518 +0000 UTC"
Sep 16 18:40:09.249676 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:09.249674    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15466h26m47.485583254s"
Sep 16 18:40:09.290852 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:09.290819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:09.291049 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:09.291021    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:09.291467 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:09.291444    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:09.291627 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:09.291555    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:09.294081 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:09.294059    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:09.294193 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:09.294176    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:09.306486 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:09.306461    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:09.345928 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:09.345837    2582 generic.go:356] "Generic (PLEG): container finished" podID="9e9ace810eda97af717471d4045e5e4a" containerID="d451e20b32e52744b63bfc0434df4257ee91d3211038cbd285f5e337ccee9581" exitCode=0
Sep 16 18:40:09.346083 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:09.346037    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal" event={"ID":"9e9ace810eda97af717471d4045e5e4a","Type":"ContainerDied","Data":"d451e20b32e52744b63bfc0434df4257ee91d3211038cbd285f5e337ccee9581"}
Sep 16 18:40:10.378289 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:10.378022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal" event={"ID":"9e9ace810eda97af717471d4045e5e4a","Type":"ContainerStarted","Data":"8799c3eecd7a179c33a71432047f5ba07b6111ef20d841937d6dc816a792a9cc"}
Sep 16 18:40:10.395203 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:10.395145    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-50.ec2.internal" podStartSLOduration=3.395125808 podStartE2EDuration="3.395125808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:10.394403962 +0000 UTC m=+5.581779753" watchObservedRunningTime="2026-09-16 18:40:10.395125808 +0000 UTC m=+5.582501599"
Sep 16 18:40:10.804587 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:10.804493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:10.804746 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:10.804602    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:10.804812 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:10.804743    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:10.804812 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:10.804806    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs podName:47c4bc73-2eff-41ea-91fc-5acce1bcd526 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:14.804786197 +0000 UTC m=+9.992161983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs") pod "network-metrics-daemon-2bzzg" (UID: "47c4bc73-2eff-41ea-91fc-5acce1bcd526") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:10.805346 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:10.805324    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:10.805453 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:10.805385    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret podName:935eb2ed-1f77-4417-ad43-71cb39d08c66 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:14.805367225 +0000 UTC m=+9.992743006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret") pod "global-pull-secret-syncer-stn2j" (UID: "935eb2ed-1f77-4417-ad43-71cb39d08c66") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:10.905028 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:10.904987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6zv\" (UniqueName: \"kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv\") pod \"network-check-target-p52xv\" (UID: \"056ad46e-0656-444d-bed0-680d3562ee88\") " pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:10.905231 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:10.905212    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:10.905300 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:10.905245    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:10.905300 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:10.905258    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df6zv for pod openshift-network-diagnostics/network-check-target-p52xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:10.905404 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:10.905316    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv podName:056ad46e-0656-444d-bed0-680d3562ee88 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:14.905296895 +0000 UTC m=+10.092672667 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-df6zv" (UniqueName: "kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv") pod "network-check-target-p52xv" (UID: "056ad46e-0656-444d-bed0-680d3562ee88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:11.292075 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:11.292042    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:11.292250 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:11.292192    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:11.292591 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:11.292571    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:11.292686 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:11.292672    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:11.292749 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:11.292728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:11.292878 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:11.292804    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:13.292033 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:13.291868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:13.292033 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:13.291887    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:13.292033 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:13.292026    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:13.292623 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:13.292079    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:13.292623 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:13.292128    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:13.292623 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:13.292182    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:14.842973 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:14.842915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:14.843397 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:14.843009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:14.843397 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:14.843175    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:14.843397 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:14.843242    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret podName:935eb2ed-1f77-4417-ad43-71cb39d08c66 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.843221818 +0000 UTC m=+18.030597609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret") pod "global-pull-secret-syncer-stn2j" (UID: "935eb2ed-1f77-4417-ad43-71cb39d08c66") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:14.843663 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:14.843645    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:14.843722 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:14.843702    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs podName:47c4bc73-2eff-41ea-91fc-5acce1bcd526 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.843684918 +0000 UTC m=+18.031060692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs") pod "network-metrics-daemon-2bzzg" (UID: "47c4bc73-2eff-41ea-91fc-5acce1bcd526") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:14.944201 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:14.944136    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6zv\" (UniqueName: \"kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv\") pod \"network-check-target-p52xv\" (UID: \"056ad46e-0656-444d-bed0-680d3562ee88\") " pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:14.944446 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:14.944281    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:14.944446 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:14.944304    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:14.944446 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:14.944318    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df6zv for pod openshift-network-diagnostics/network-check-target-p52xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:14.944446 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:14.944386    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv podName:056ad46e-0656-444d-bed0-680d3562ee88 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.944368067 +0000 UTC m=+18.131743853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-df6zv" (UniqueName: "kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv") pod "network-check-target-p52xv" (UID: "056ad46e-0656-444d-bed0-680d3562ee88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:15.293606 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:15.293572    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:15.293802 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:15.293709    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:15.294280 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:15.294258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:15.294446 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:15.294410    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:15.294515 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:15.294491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:15.294617 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:15.294600    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:17.294920 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:17.294883    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:17.295408 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:17.295046    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:17.295481 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:17.295416    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:17.295545 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:17.295509    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:17.295602 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:17.295556    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:17.295643 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:17.295622    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:19.291600 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:19.291569    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:19.291600 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:19.291582    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:19.291600 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:19.291593    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:19.292184 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:19.291686    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:19.292184 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:19.291750    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:19.292184 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:19.291803    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:21.291344 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:21.291298    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:21.291344 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:21.291345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:21.291834 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:21.291299    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:21.291834 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:21.291433    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:21.291834 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:21.291498    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:21.291834 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:21.291582    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:22.908052 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:22.908019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:22.908470 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:22.908063    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:22.908470 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:22.908159    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:22.908470 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:22.908174    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:22.908470 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:22.908205    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret podName:935eb2ed-1f77-4417-ad43-71cb39d08c66 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:38.908191736 +0000 UTC m=+34.095567500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret") pod "global-pull-secret-syncer-stn2j" (UID: "935eb2ed-1f77-4417-ad43-71cb39d08c66") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:22.908470 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:22.908235    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs podName:47c4bc73-2eff-41ea-91fc-5acce1bcd526 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:38.908216819 +0000 UTC m=+34.095592591 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs") pod "network-metrics-daemon-2bzzg" (UID: "47c4bc73-2eff-41ea-91fc-5acce1bcd526") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:23.008717 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:23.008690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6zv\" (UniqueName: \"kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv\") pod \"network-check-target-p52xv\" (UID: \"056ad46e-0656-444d-bed0-680d3562ee88\") " pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:23.008882 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:23.008855    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:23.008882 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:23.008882    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:23.009080 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:23.008894    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df6zv for pod openshift-network-diagnostics/network-check-target-p52xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:23.009080 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:23.008955    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv podName:056ad46e-0656-444d-bed0-680d3562ee88 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:39.008943306 +0000 UTC m=+34.196319084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-df6zv" (UniqueName: "kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv") pod "network-check-target-p52xv" (UID: "056ad46e-0656-444d-bed0-680d3562ee88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:23.291584 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:23.291551    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:23.291749 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:23.291552    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:23.291749 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:23.291560    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:23.291749 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:23.291693    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:23.291941 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:23.291775    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:23.291941 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:23.291883    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:25.291878 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:25.291845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:25.292343 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:25.291971    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:25.292343 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:25.291989    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:25.292343 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:25.292091    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:25.292343 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:25.292143    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:25.292343 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:25.292212    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:27.291438 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.291213    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:27.292086 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.291213    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:27.292086 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:27.291552    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:27.292086 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.291270    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:27.292086 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:27.291599    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:27.292086 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:27.291692    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:27.410891 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.410824    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bjbtm" event={"ID":"10013ea8-6367-4299-ae77-de784d9ac6cc","Type":"ContainerStarted","Data":"a856bcc043eb706a396f52289233e038fb854f441d20ea8b5485a7a802dcac44"}
Sep 16 18:40:27.412419 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.412397    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z5pl8" event={"ID":"f8eeb34d-7a85-40ac-8755-be657c455ec6","Type":"ContainerStarted","Data":"6e5feeaccf5378f1518ee33651783d53cbeb653128b7c3eefc56427b7c3b10dd"}
Sep 16 18:40:27.414085 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.414057    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m8d2k" event={"ID":"54b82fdf-4c88-44a6-a6ac-f37f8de64fe8","Type":"ContainerStarted","Data":"c162dda40a3dff110f19625a8dc91c7224a0f98323cc4d06638d70686880b392"}
Sep 16 18:40:27.415661 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.415636    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-shplt" event={"ID":"d37b1617-d4af-42a9-8795-2074cbf8be05","Type":"ContainerStarted","Data":"9f7d52f043ed50e38c9590d6635529fe7113766da9e475d3b0c8f46e9201eb87"}
Sep 16 18:40:27.417342 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.417302    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7" event={"ID":"5a0a56bd-3ee3-41f4-9730-61b055862970","Type":"ContainerStarted","Data":"f3e33ee10136a65941672c233b59e9a40260516ee54da6fcf1789bd2be116176"}
Sep 16 18:40:27.419069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.419044    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x4jf2" event={"ID":"dd12d598-68f2-4545-a589-be29dda56485","Type":"ContainerStarted","Data":"712c8268269b028f4ced2b5e227a60bd212736f92a80f0e26da68c0c4203297f"}
Sep 16 18:40:27.422597 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.422569    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8" event={"ID":"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098","Type":"ContainerStarted","Data":"29fa99fa8ddf414c2e500b8a3b9ad1550bfcc8263cd6a162dc5a25b07d8f40c7"}
Sep 16 18:40:27.422698 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.422642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8" event={"ID":"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098","Type":"ContainerStarted","Data":"e2288630561ae6e26decd2f784c616d6f1e2c3b14aed78696d0d38acc05efcf3"}
Sep 16 18:40:27.422698 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.422691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8" event={"ID":"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098","Type":"ContainerStarted","Data":"153eccd7c1a7d8bfb2ad81d5d617b549fc8da5ed0efd59d298e22a496049b91c"}
Sep 16 18:40:27.422793 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.422742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8" event={"ID":"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098","Type":"ContainerStarted","Data":"7085d11e10709ab184003d538a0d455ea6181ee0447e18088d66bca16bb6782f"}
Sep 16 18:40:27.422835 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.422791    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8" event={"ID":"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098","Type":"ContainerStarted","Data":"65742486d32c2c7f894916b39145ed467e007aa0b115fa6ad0c9a264e32ea79e"}
Sep 16 18:40:27.422882 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.422840    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8" event={"ID":"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098","Type":"ContainerStarted","Data":"dcf8698969dafad6c472e622a6287f685809967a3d88be4a782fc6fd0307d09c"}
Sep 16 18:40:27.424238 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.424218    2582 generic.go:356] "Generic (PLEG): container finished" podID="78095364-693e-4375-ae1e-cd23e8a6001d" containerID="49a7abee7d88bf5c2e0642cc642aaf81ca0be21e241829db393e0546e478c6d6" exitCode=0
Sep 16 18:40:27.424351 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.424249    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnnjq" event={"ID":"78095364-693e-4375-ae1e-cd23e8a6001d","Type":"ContainerDied","Data":"49a7abee7d88bf5c2e0642cc642aaf81ca0be21e241829db393e0546e478c6d6"}
Sep 16 18:40:27.433961 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.433921    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bjbtm" podStartSLOduration=10.08714438 podStartE2EDuration="21.433885549s" totalImagesPullingTime="11.346741169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:27.433315107 +0000 UTC m=+22.620690895" watchObservedRunningTime="2026-09-16 18:40:27.433885549 +0000 UTC m=+22.621261338"
Sep 16 18:40:27.451013 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.450963    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-shplt" podStartSLOduration=2.974858787 podStartE2EDuration="21.45094666s" totalImagesPullingTime="18.476087873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:27.450410361 +0000 UTC m=+22.637786150" watchObservedRunningTime="2026-09-16 18:40:27.45094666 +0000 UTC m=+22.638322444"
Sep 16 18:40:27.483844 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.483786    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x4jf2" podStartSLOduration=2.962871286 podStartE2EDuration="21.483774143s" totalImagesPullingTime="18.520902857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:27.483719853 +0000 UTC m=+22.671095636" watchObservedRunningTime="2026-09-16 18:40:27.483774143 +0000 UTC m=+22.671149907"
Sep 16 18:40:27.513636 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.513601    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m8d2k" podStartSLOduration=3.042973845 podStartE2EDuration="21.513589719s" totalImagesPullingTime="18.470615874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:27.513180697 +0000 UTC m=+22.700556484" watchObservedRunningTime="2026-09-16 18:40:27.513589719 +0000 UTC m=+22.700965505"
Sep 16 18:40:27.536300 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:27.536267    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z5pl8" podStartSLOduration=3.055137939 podStartE2EDuration="21.536257681s" totalImagesPullingTime="18.481119742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:27.536106067 +0000 UTC m=+22.723481854" watchObservedRunningTime="2026-09-16 18:40:27.536257681 +0000 UTC m=+22.723633468"
Sep 16 18:40:28.276363 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:28.276219    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:40:28.428093 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:28.428059    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pkx6x" event={"ID":"f54367c0-5c70-4b72-9a0f-ee5182e7e088","Type":"ContainerStarted","Data":"8c1533b3056c6b8f3b7dd95dc915ceaa457f087ce201aa35d387a275028b3850"}
Sep 16 18:40:28.430421 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:28.430350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7" event={"ID":"5a0a56bd-3ee3-41f4-9730-61b055862970","Type":"ContainerStarted","Data":"d5c3dea9ddb7c0ff2037a7648b45687efbfbf525df54cb64b0b1d4ed03960e2c"}
Sep 16 18:40:28.448655 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:28.448611    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pkx6x" podStartSLOduration=3.98478436 podStartE2EDuration="22.448591419s" totalImagesPullingTime="18.463807059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:28.448097249 +0000 UTC m=+23.635473036" watchObservedRunningTime="2026-09-16 18:40:28.448591419 +0000 UTC m=+23.635967207"
Sep 16 18:40:29.237794 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:29.237697    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:40:28.276362009Z","UUID":"517ce2c5-2d7d-4819-b088-bb7b077ca768","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:40:29.239271 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:29.239250    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:40:29.239271 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:29.239279    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:40:29.291077 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:29.291054    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:29.291275 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:29.291169    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:29.291349 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:29.291248    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:29.291406 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:29.291387    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:29.291457 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:29.291421    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:29.291549 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:29.291527    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:29.436657 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:29.436627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8" event={"ID":"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098","Type":"ContainerStarted","Data":"1b8ce0eca0c0225c1d12b553aba8582a3433f4033ed050d97d940010d02a1ad5"}
Sep 16 18:40:29.439174 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:29.439152    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7" event={"ID":"5a0a56bd-3ee3-41f4-9730-61b055862970","Type":"ContainerStarted","Data":"3d6e4f6c5d3e219fe00736bb2bc3c174a3c4adb964fb81d5bd3e020295ea16c4"}
Sep 16 18:40:30.670425 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:30.670390    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bjbtm"
Sep 16 18:40:30.670986 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:30.670968    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bjbtm"
Sep 16 18:40:30.692826 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:30.692776    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8kqb7" podStartSLOduration=4.001962994 podStartE2EDuration="24.692763171s" totalImagesPullingTime="20.690800177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:29.46723402 +0000 UTC m=+24.654609811" watchObservedRunningTime="2026-09-16 18:40:30.692763171 +0000 UTC m=+25.880138959"
Sep 16 18:40:31.291256 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:31.291227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:31.291256 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:31.291242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:31.291476 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:31.291326    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:31.291476 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:31.291426    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:31.291476 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:31.291444    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:31.291578 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:31.291518    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:31.443352 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:31.443323    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bjbtm"
Sep 16 18:40:31.443855 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:31.443832    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bjbtm"
Sep 16 18:40:32.449874 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:32.449624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8" event={"ID":"6d6c78ec-8ef8-4148-bff5-c7d9e7aa2098","Type":"ContainerStarted","Data":"3a92094bc3bf59d538ed23a42e7c413d358786d927ed3549cfd58395e4eee8a2"}
Sep 16 18:40:32.450503 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:32.449954    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:32.451848 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:32.451824    2582 generic.go:356] "Generic (PLEG): container finished" podID="78095364-693e-4375-ae1e-cd23e8a6001d" containerID="40e8c772ba28a3c55e7ce1e8a453ce21c5c42ad4e3fc24ad0f78dde7651d42f5" exitCode=0
Sep 16 18:40:32.451988 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:32.451887    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnnjq" event={"ID":"78095364-693e-4375-ae1e-cd23e8a6001d","Type":"ContainerDied","Data":"40e8c772ba28a3c55e7ce1e8a453ce21c5c42ad4e3fc24ad0f78dde7651d42f5"}
Sep 16 18:40:32.466137 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:32.466118    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:32.486232 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:32.486197    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8" podStartSLOduration=7.969089335 podStartE2EDuration="26.486184928s" totalImagesPullingTime="18.517095593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:32.485956628 +0000 UTC m=+27.673332414" watchObservedRunningTime="2026-09-16 18:40:32.486184928 +0000 UTC m=+27.673560714"
Sep 16 18:40:33.290960 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:33.290933    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:33.290960 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:33.290947    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:33.291269 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:33.291034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:33.291269 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:33.291050    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:33.291269 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:33.291129    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:33.291269 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:33.291230    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:33.454565 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:33.454536    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:33.455037 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:33.454941    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:33.469302 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:33.469278    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:40:33.822224 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:33.822017    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-stn2j"]
Sep 16 18:40:33.822346 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:33.822257    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:33.822414 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:33.822365    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:33.825989 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:33.825964    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2bzzg"]
Sep 16 18:40:33.826083 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:33.826059    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:33.826149 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:33.826134    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:33.841983 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:33.841959    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p52xv"]
Sep 16 18:40:33.842064 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:33.842040    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:33.842156 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:33.842136    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:34.459042 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:34.459003    2582 generic.go:356] "Generic (PLEG): container finished" podID="78095364-693e-4375-ae1e-cd23e8a6001d" containerID="1bb5aa26a1ec271407aefae0957bf2ab5b9f4ad7741de5fbc5de752370e6625d" exitCode=0
Sep 16 18:40:34.459546 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:34.459106    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnnjq" event={"ID":"78095364-693e-4375-ae1e-cd23e8a6001d","Type":"ContainerDied","Data":"1bb5aa26a1ec271407aefae0957bf2ab5b9f4ad7741de5fbc5de752370e6625d"}
Sep 16 18:40:35.292264 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:35.292236    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:35.292429 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:35.292316    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:35.292549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:35.292525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:35.292692 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:35.292572    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:35.292692 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:35.292655    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:35.292774 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:35.292724    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:36.466782 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:36.466753    2582 generic.go:356] "Generic (PLEG): container finished" podID="78095364-693e-4375-ae1e-cd23e8a6001d" containerID="73e4f0f211357e66f63af81e01fbc49f753218b51499096fa96c62a3862572e5" exitCode=0
Sep 16 18:40:36.467154 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:36.466800    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnnjq" event={"ID":"78095364-693e-4375-ae1e-cd23e8a6001d","Type":"ContainerDied","Data":"73e4f0f211357e66f63af81e01fbc49f753218b51499096fa96c62a3862572e5"}
Sep 16 18:40:37.291779 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:37.291746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:37.291975 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:37.291746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:37.291975 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:37.291879    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:37.291975 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:37.291890    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:37.292142 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:37.291980    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:37.292142 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:37.292068    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:38.935418 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:38.935385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:38.935992 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:38.935440    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:38.935992 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:38.935546    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:38.935992 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:38.935550    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:38.935992 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:38.935618    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs podName:47c4bc73-2eff-41ea-91fc-5acce1bcd526 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:10.935596336 +0000 UTC m=+66.122972115 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs") pod "network-metrics-daemon-2bzzg" (UID: "47c4bc73-2eff-41ea-91fc-5acce1bcd526") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:38.935992 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:38.935637    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret podName:935eb2ed-1f77-4417-ad43-71cb39d08c66 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:10.935627875 +0000 UTC m=+66.123003641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret") pod "global-pull-secret-syncer-stn2j" (UID: "935eb2ed-1f77-4417-ad43-71cb39d08c66") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:39.036091 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.036059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6zv\" (UniqueName: \"kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv\") pod \"network-check-target-p52xv\" (UID: \"056ad46e-0656-444d-bed0-680d3562ee88\") " pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:39.036255 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:39.036234    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:39.036316 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:39.036259    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:39.036316 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:39.036275    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df6zv for pod openshift-network-diagnostics/network-check-target-p52xv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:39.036399 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:39.036330    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv podName:056ad46e-0656-444d-bed0-680d3562ee88 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:11.036313851 +0000 UTC m=+66.223689623 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-df6zv" (UniqueName: "kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv") pod "network-check-target-p52xv" (UID: "056ad46e-0656-444d-bed0-680d3562ee88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:39.291012 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.290987    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:39.291165 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.291018    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:39.291165 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.290991    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:39.291165 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:39.291099    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stn2j" podUID="935eb2ed-1f77-4417-ad43-71cb39d08c66"
Sep 16 18:40:39.291328 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:39.291198    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p52xv" podUID="056ad46e-0656-444d-bed0-680d3562ee88"
Sep 16 18:40:39.291328 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:39.291287    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:40:39.577618 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.577592    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-50.ec2.internal" event="NodeReady"
Sep 16 18:40:39.577775 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.577684    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:40:39.665901 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.665872    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hl5jx"]
Sep 16 18:40:39.670577 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.670552    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:39.671446 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.671416    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xlrfw"]
Sep 16 18:40:39.674752 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.674731    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xlrfw"
Sep 16 18:40:39.677114 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.677094    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:40:39.677114 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.677109    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:40:39.680482 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.680338    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xfn4n\""
Sep 16 18:40:39.684846 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.684823    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:40:39.685833 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.685773    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:40:39.686017 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.685998    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:40:39.686238 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.686220    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mg98h\""
Sep 16 18:40:39.690016 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.689995    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xlrfw"]
Sep 16 18:40:39.701021 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.700998    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hl5jx"]
Sep 16 18:40:39.843966 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.843838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prqp6\" (UniqueName: \"kubernetes.io/projected/a00e8866-0ae0-4245-b716-faac4ee1cd3f-kube-api-access-prqp6\") pod \"ingress-canary-xlrfw\" (UID: \"a00e8866-0ae0-4245-b716-faac4ee1cd3f\") " pod="openshift-ingress-canary/ingress-canary-xlrfw"
Sep 16 18:40:39.844118 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.843973    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a00e8866-0ae0-4245-b716-faac4ee1cd3f-cert\") pod \"ingress-canary-xlrfw\" (UID: \"a00e8866-0ae0-4245-b716-faac4ee1cd3f\") " pod="openshift-ingress-canary/ingress-canary-xlrfw"
Sep 16 18:40:39.844118 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.844069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a673e9f7-670e-4604-95f2-7aab6f511727-config-volume\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:39.844118 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.844095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a673e9f7-670e-4604-95f2-7aab6f511727-tmp-dir\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:39.844278 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.844118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwrdd\" (UniqueName: \"kubernetes.io/projected/a673e9f7-670e-4604-95f2-7aab6f511727-kube-api-access-hwrdd\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:39.844278 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.844175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:39.944919 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.944872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a00e8866-0ae0-4245-b716-faac4ee1cd3f-cert\") pod \"ingress-canary-xlrfw\" (UID: \"a00e8866-0ae0-4245-b716-faac4ee1cd3f\") " pod="openshift-ingress-canary/ingress-canary-xlrfw"
Sep 16 18:40:39.945348 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.944965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a673e9f7-670e-4604-95f2-7aab6f511727-config-volume\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:39.945348 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.944990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a673e9f7-670e-4604-95f2-7aab6f511727-tmp-dir\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:39.945348 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.945013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwrdd\" (UniqueName: \"kubernetes.io/projected/a673e9f7-670e-4604-95f2-7aab6f511727-kube-api-access-hwrdd\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:39.945348 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.945198    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:39.945348 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.945244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prqp6\" (UniqueName: \"kubernetes.io/projected/a00e8866-0ae0-4245-b716-faac4ee1cd3f-kube-api-access-prqp6\") pod \"ingress-canary-xlrfw\" (UID: \"a00e8866-0ae0-4245-b716-faac4ee1cd3f\") " pod="openshift-ingress-canary/ingress-canary-xlrfw"
Sep 16 18:40:39.945605 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:39.945362    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:39.945605 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.945404    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a673e9f7-670e-4604-95f2-7aab6f511727-tmp-dir\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:39.945605 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:39.945437    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls podName:a673e9f7-670e-4604-95f2-7aab6f511727 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:40.445420472 +0000 UTC m=+35.632796237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls") pod "dns-default-hl5jx" (UID: "a673e9f7-670e-4604-95f2-7aab6f511727") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:39.945750 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.945648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a673e9f7-670e-4604-95f2-7aab6f511727-config-volume\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:39.950180 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.950048    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a00e8866-0ae0-4245-b716-faac4ee1cd3f-cert\") pod \"ingress-canary-xlrfw\" (UID: \"a00e8866-0ae0-4245-b716-faac4ee1cd3f\") " pod="openshift-ingress-canary/ingress-canary-xlrfw"
Sep 16 18:40:39.961927 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.961886    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwrdd\" (UniqueName: \"kubernetes.io/projected/a673e9f7-670e-4604-95f2-7aab6f511727-kube-api-access-hwrdd\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:39.962286 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.962266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prqp6\" (UniqueName: \"kubernetes.io/projected/a00e8866-0ae0-4245-b716-faac4ee1cd3f-kube-api-access-prqp6\") pod \"ingress-canary-xlrfw\" (UID: \"a00e8866-0ae0-4245-b716-faac4ee1cd3f\") " pod="openshift-ingress-canary/ingress-canary-xlrfw"
Sep 16 18:40:39.989005 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:39.988981    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xlrfw"
Sep 16 18:40:40.448414 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:40.448378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:40.448546 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:40.448527    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:40.448612 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:40.448600    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls podName:a673e9f7-670e-4604-95f2-7aab6f511727 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:41.448579835 +0000 UTC m=+36.635955615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls") pod "dns-default-hl5jx" (UID: "a673e9f7-670e-4604-95f2-7aab6f511727") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:40.478304 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:40.478276    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnnjq" event={"ID":"78095364-693e-4375-ae1e-cd23e8a6001d","Type":"ContainerStarted","Data":"545d0ae9bc5320d4ef674915200d3c7fad6689d1f38875131554c1cc5703a928"}
Sep 16 18:40:40.590735 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:40.590714    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xlrfw"]
Sep 16 18:40:40.601742 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:40:40.601719    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00e8866_0ae0_4245_b716_faac4ee1cd3f.slice/crio-f144aade8b062d74449aefeb9ee219d7efbaa990ff2170621d8d9db176dc8a83 WatchSource:0}: Error finding container f144aade8b062d74449aefeb9ee219d7efbaa990ff2170621d8d9db176dc8a83: Status 404 returned error can't find the container with id f144aade8b062d74449aefeb9ee219d7efbaa990ff2170621d8d9db176dc8a83
Sep 16 18:40:41.291784 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.291757    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:40:41.292472 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.291825    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:40:41.292472 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.291827    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:40:41.294857 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.294836    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:40:41.295654 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.295634    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-564xp\""
Sep 16 18:40:41.296002 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.295983    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:40:41.296119 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.296101    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:40:41.296228 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.296208    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z52vg\""
Sep 16 18:40:41.296321 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.296229    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:40:41.456638 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.456612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:41.456779 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:41.456762    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:41.456842 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:41.456831    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls podName:a673e9f7-670e-4604-95f2-7aab6f511727 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:43.456814596 +0000 UTC m=+38.644190378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls") pod "dns-default-hl5jx" (UID: "a673e9f7-670e-4604-95f2-7aab6f511727") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:41.483539 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.483499    2582 generic.go:356] "Generic (PLEG): container finished" podID="78095364-693e-4375-ae1e-cd23e8a6001d" containerID="545d0ae9bc5320d4ef674915200d3c7fad6689d1f38875131554c1cc5703a928" exitCode=0
Sep 16 18:40:41.483539 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.483534    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnnjq" event={"ID":"78095364-693e-4375-ae1e-cd23e8a6001d","Type":"ContainerDied","Data":"545d0ae9bc5320d4ef674915200d3c7fad6689d1f38875131554c1cc5703a928"}
Sep 16 18:40:41.484962 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:41.484935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xlrfw" event={"ID":"a00e8866-0ae0-4245-b716-faac4ee1cd3f","Type":"ContainerStarted","Data":"f144aade8b062d74449aefeb9ee219d7efbaa990ff2170621d8d9db176dc8a83"}
Sep 16 18:40:42.492838 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:42.492806    2582 generic.go:356] "Generic (PLEG): container finished" podID="78095364-693e-4375-ae1e-cd23e8a6001d" containerID="a2bdc402d36a11e7969c729d70dfc580388a8a5c4dfdcb66de21e80b8a8ff8aa" exitCode=0
Sep 16 18:40:42.493320 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:42.492860    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnnjq" event={"ID":"78095364-693e-4375-ae1e-cd23e8a6001d","Type":"ContainerDied","Data":"a2bdc402d36a11e7969c729d70dfc580388a8a5c4dfdcb66de21e80b8a8ff8aa"}
Sep 16 18:40:43.473054 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:43.472883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:43.473179 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:43.473000    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:43.473179 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:43.473134    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls podName:a673e9f7-670e-4604-95f2-7aab6f511727 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:47.473120102 +0000 UTC m=+42.660495866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls") pod "dns-default-hl5jx" (UID: "a673e9f7-670e-4604-95f2-7aab6f511727") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:43.498374 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:43.498348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnnjq" event={"ID":"78095364-693e-4375-ae1e-cd23e8a6001d","Type":"ContainerStarted","Data":"2a3b82b7cb5fb840a360728e642d93b55765e21aafeb40670eadd00a594df6c0"}
Sep 16 18:40:43.499810 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:43.499789    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xlrfw" event={"ID":"a00e8866-0ae0-4245-b716-faac4ee1cd3f","Type":"ContainerStarted","Data":"4c9fb86a12e218825bd2a80ca21ccf20ed1e5ae7f14b63091522655ffdaf4e77"}
Sep 16 18:40:43.525640 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:43.525596    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gnnjq" podStartSLOduration=8.613621359 podStartE2EDuration="37.525584726s" totalImagesPullingTime="28.911963367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:43.524714335 +0000 UTC m=+38.712090122" watchObservedRunningTime="2026-09-16 18:40:43.525584726 +0000 UTC m=+38.712960512"
Sep 16 18:40:43.544516 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:43.544473    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xlrfw" podStartSLOduration=2.277320793 podStartE2EDuration="4.544461804s" totalImagesPullingTime="2.267141011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:43.543694122 +0000 UTC m=+38.731069913" watchObservedRunningTime="2026-09-16 18:40:43.544461804 +0000 UTC m=+38.731837587"
Sep 16 18:40:47.501322 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:47.501282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:47.501751 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:47.501426    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:47.501751 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:47.501488    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls podName:a673e9f7-670e-4604-95f2-7aab6f511727 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:55.50146955 +0000 UTC m=+50.688845388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls") pod "dns-default-hl5jx" (UID: "a673e9f7-670e-4604-95f2-7aab6f511727") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:55.555229 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:40:55.555193    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:40:55.555678 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:55.555321    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:55.555678 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:40:55.555373    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls podName:a673e9f7-670e-4604-95f2-7aab6f511727 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:11.555359151 +0000 UTC m=+66.742734917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls") pod "dns-default-hl5jx" (UID: "a673e9f7-670e-4604-95f2-7aab6f511727") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:05.473852 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:05.473823    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxbd8"
Sep 16 18:41:10.956062 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:10.956024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:41:10.956646 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:10.956087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:41:10.959847 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:10.959826    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:10.959900 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:10.959831    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:10.966492 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:41:10.966469    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:41:10.966611 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:41:10.966532    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs podName:47c4bc73-2eff-41ea-91fc-5acce1bcd526 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:14.966514154 +0000 UTC m=+130.153889919 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs") pod "network-metrics-daemon-2bzzg" (UID: "47c4bc73-2eff-41ea-91fc-5acce1bcd526") : secret "metrics-daemon-secret" not found
Sep 16 18:41:10.970221 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:10.970194    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/935eb2ed-1f77-4417-ad43-71cb39d08c66-original-pull-secret\") pod \"global-pull-secret-syncer-stn2j\" (UID: \"935eb2ed-1f77-4417-ad43-71cb39d08c66\") " pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:41:11.010522 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:11.010496    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stn2j"
Sep 16 18:41:11.056967 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:11.056938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6zv\" (UniqueName: \"kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv\") pod \"network-check-target-p52xv\" (UID: \"056ad46e-0656-444d-bed0-680d3562ee88\") " pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:41:11.061055 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:11.061033    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:11.071216 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:11.071197    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:11.081525 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:11.081500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6zv\" (UniqueName: \"kubernetes.io/projected/056ad46e-0656-444d-bed0-680d3562ee88-kube-api-access-df6zv\") pod \"network-check-target-p52xv\" (UID: \"056ad46e-0656-444d-bed0-680d3562ee88\") " pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:41:11.322325 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:11.322270    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z52vg\""
Sep 16 18:41:11.327594 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:11.327576    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:41:11.333694 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:11.333580    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-stn2j"]
Sep 16 18:41:11.336333 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:41:11.336312    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935eb2ed_1f77_4417_ad43_71cb39d08c66.slice/crio-39041ca04f025f0ef255bd9cdb359345fe9e050a37e3fb22092675c646a1d4d6 WatchSource:0}: Error finding container 39041ca04f025f0ef255bd9cdb359345fe9e050a37e3fb22092675c646a1d4d6: Status 404 returned error can't find the container with id 39041ca04f025f0ef255bd9cdb359345fe9e050a37e3fb22092675c646a1d4d6
Sep 16 18:41:11.560882 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:11.560856    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:41:11.561056 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:41:11.561028    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:11.561133 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:41:11.561118    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls podName:a673e9f7-670e-4604-95f2-7aab6f511727 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:43.561092809 +0000 UTC m=+98.748468594 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls") pod "dns-default-hl5jx" (UID: "a673e9f7-670e-4604-95f2-7aab6f511727") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:11.564139 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:11.564110    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-stn2j" event={"ID":"935eb2ed-1f77-4417-ad43-71cb39d08c66","Type":"ContainerStarted","Data":"39041ca04f025f0ef255bd9cdb359345fe9e050a37e3fb22092675c646a1d4d6"}
Sep 16 18:41:11.641862 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:11.641807    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p52xv"]
Sep 16 18:41:11.645179 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:41:11.645158    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056ad46e_0656_444d_bed0_680d3562ee88.slice/crio-ad298240052f9e19e16c25c200257a1aee330bfe75de0daf6e641b5ee0f9825a WatchSource:0}: Error finding container ad298240052f9e19e16c25c200257a1aee330bfe75de0daf6e641b5ee0f9825a: Status 404 returned error can't find the container with id ad298240052f9e19e16c25c200257a1aee330bfe75de0daf6e641b5ee0f9825a
Sep 16 18:41:12.568948 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:12.568895    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p52xv" event={"ID":"056ad46e-0656-444d-bed0-680d3562ee88","Type":"ContainerStarted","Data":"ad298240052f9e19e16c25c200257a1aee330bfe75de0daf6e641b5ee0f9825a"}
Sep 16 18:41:16.581890 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:16.581844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-stn2j" event={"ID":"935eb2ed-1f77-4417-ad43-71cb39d08c66","Type":"ContainerStarted","Data":"29f02f4a79412200414a429f7b32ff3fc6a14f8222bf32d82aede3bb151ab207"}
Sep 16 18:41:16.583876 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:16.583846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p52xv" event={"ID":"056ad46e-0656-444d-bed0-680d3562ee88","Type":"ContainerStarted","Data":"2f772b73df1aa711ac22116c2a8f4757faadeb70e231903ddef78f53079e8050"}
Sep 16 18:41:16.584036 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:16.583958    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:41:16.615796 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:16.615751    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-stn2j" podStartSLOduration=66.032835047 podStartE2EDuration="1m10.615736216s" totalImagesPullingTime="4.582901169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:16.612525677 +0000 UTC m=+71.799901483" watchObservedRunningTime="2026-09-16 18:41:16.615736216 +0000 UTC m=+71.803112074"
Sep 16 18:41:16.644163 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:16.644114    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p52xv" podStartSLOduration=66.374400391 podStartE2EDuration="1m10.644099034s" totalImagesPullingTime="4.269698643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:16.642851936 +0000 UTC m=+71.830227724" watchObservedRunningTime="2026-09-16 18:41:16.644099034 +0000 UTC m=+71.831474820"
Sep 16 18:41:43.567793 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:43.567754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:41:43.568221 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:41:43.567923    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:43.568221 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:41:43.568002    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls podName:a673e9f7-670e-4604-95f2-7aab6f511727 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:47.567986344 +0000 UTC m=+162.755362108 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls") pod "dns-default-hl5jx" (UID: "a673e9f7-670e-4604-95f2-7aab6f511727") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:47.589650 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:41:47.589615    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p52xv"
Sep 16 18:42:13.375329 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.375293    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-sjlt4"]
Sep 16 18:42:13.377087 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.377068    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.379603 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.379569    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:42:13.380092 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.380071    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:42:13.381095 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.381070    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:42:13.381752 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.381733    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:42:13.382972 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.382945    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g76l4\""
Sep 16 18:42:13.387560 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.387540    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:42:13.398122 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.398098    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-sjlt4"]
Sep 16 18:42:13.450335 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.450310    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aca151a-3031-429f-aa10-fb8d21ba5a27-serving-cert\") pod \"console-operator-696884fb9c-sjlt4\" (UID: \"3aca151a-3031-429f-aa10-fb8d21ba5a27\") " pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.450424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.450347    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3aca151a-3031-429f-aa10-fb8d21ba5a27-config\") pod \"console-operator-696884fb9c-sjlt4\" (UID: \"3aca151a-3031-429f-aa10-fb8d21ba5a27\") " pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.450424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.450370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlg6z\" (UniqueName: \"kubernetes.io/projected/3aca151a-3031-429f-aa10-fb8d21ba5a27-kube-api-access-rlg6z\") pod \"console-operator-696884fb9c-sjlt4\" (UID: \"3aca151a-3031-429f-aa10-fb8d21ba5a27\") " pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.450424 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.450393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3aca151a-3031-429f-aa10-fb8d21ba5a27-trusted-ca\") pod \"console-operator-696884fb9c-sjlt4\" (UID: \"3aca151a-3031-429f-aa10-fb8d21ba5a27\") " pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.476587 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.476566    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dd47fb98c-2v75m"]
Sep 16 18:42:13.478219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.478204    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.483421 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.483391    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:42:13.483971 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.483947    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:42:13.484084 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.483973    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:42:13.484316 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.484114    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gq4nt\""
Sep 16 18:42:13.485442 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.485418    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z6q5q"]
Sep 16 18:42:13.487886 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.487860    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z6q5q"
Sep 16 18:42:13.490394 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.490373    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8b6pg\""
Sep 16 18:42:13.490705 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.490683    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:42:13.490959 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.490896    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 18:42:13.493321 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.493296    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:42:13.494300 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.494278    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd47fb98c-2v75m"]
Sep 16 18:42:13.498498 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.498478    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z6q5q"]
Sep 16 18:42:13.551609 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.551574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d78a830a-17d2-4940-be08-149b0b789bbb-trusted-ca\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.551702 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.551618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlg6z\" (UniqueName: \"kubernetes.io/projected/3aca151a-3031-429f-aa10-fb8d21ba5a27-kube-api-access-rlg6z\") pod \"console-operator-696884fb9c-sjlt4\" (UID: \"3aca151a-3031-429f-aa10-fb8d21ba5a27\") " pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.551702 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.551640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d78a830a-17d2-4940-be08-149b0b789bbb-image-registry-private-configuration\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.551702 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.551655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mgcv\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-kube-api-access-6mgcv\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.551702 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.551673    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3aca151a-3031-429f-aa10-fb8d21ba5a27-trusted-ca\") pod \"console-operator-696884fb9c-sjlt4\" (UID: \"3aca151a-3031-429f-aa10-fb8d21ba5a27\") " pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.551702 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.551701    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-bound-sa-token\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.551937 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.551719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d78a830a-17d2-4940-be08-149b0b789bbb-installation-pull-secrets\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.551937 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.551739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aca151a-3031-429f-aa10-fb8d21ba5a27-serving-cert\") pod \"console-operator-696884fb9c-sjlt4\" (UID: \"3aca151a-3031-429f-aa10-fb8d21ba5a27\") " pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.551937 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.551763    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.551937 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.551786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d78a830a-17d2-4940-be08-149b0b789bbb-registry-certificates\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.551937 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.551819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3aca151a-3031-429f-aa10-fb8d21ba5a27-config\") pod \"console-operator-696884fb9c-sjlt4\" (UID: \"3aca151a-3031-429f-aa10-fb8d21ba5a27\") " pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.551937 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.551840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d78a830a-17d2-4940-be08-149b0b789bbb-ca-trust-extracted\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.552380 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.552354    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3aca151a-3031-429f-aa10-fb8d21ba5a27-config\") pod \"console-operator-696884fb9c-sjlt4\" (UID: \"3aca151a-3031-429f-aa10-fb8d21ba5a27\") " pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.552577 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.552562    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3aca151a-3031-429f-aa10-fb8d21ba5a27-trusted-ca\") pod \"console-operator-696884fb9c-sjlt4\" (UID: \"3aca151a-3031-429f-aa10-fb8d21ba5a27\") " pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.554037 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.554018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aca151a-3031-429f-aa10-fb8d21ba5a27-serving-cert\") pod \"console-operator-696884fb9c-sjlt4\" (UID: \"3aca151a-3031-429f-aa10-fb8d21ba5a27\") " pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.566290 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.566270    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlg6z\" (UniqueName: \"kubernetes.io/projected/3aca151a-3031-429f-aa10-fb8d21ba5a27-kube-api-access-rlg6z\") pod \"console-operator-696884fb9c-sjlt4\" (UID: \"3aca151a-3031-429f-aa10-fb8d21ba5a27\") " pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.652112 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.652060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d78a830a-17d2-4940-be08-149b0b789bbb-trusted-ca\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.652112 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.652099    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d78a830a-17d2-4940-be08-149b0b789bbb-image-registry-private-configuration\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.652255 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.652126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mgcv\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-kube-api-access-6mgcv\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.652316 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.652268    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-bound-sa-token\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.652370 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.652311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d78a830a-17d2-4940-be08-149b0b789bbb-installation-pull-secrets\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.652464 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.652443    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vps\" (UniqueName: \"kubernetes.io/projected/049b76d8-852b-42f0-9720-d8bb5e59ff5f-kube-api-access-q5vps\") pod \"volume-data-source-validator-86695b7cc7-z6q5q\" (UID: \"049b76d8-852b-42f0-9720-d8bb5e59ff5f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z6q5q"
Sep 16 18:42:13.652528 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.652497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.652583 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.652526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d78a830a-17d2-4940-be08-149b0b789bbb-registry-certificates\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.652583 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.652565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d78a830a-17d2-4940-be08-149b0b789bbb-ca-trust-extracted\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.652689 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:13.652606    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:42:13.652689 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:13.652622    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd47fb98c-2v75m: secret "image-registry-tls" not found
Sep 16 18:42:13.652689 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:13.652684    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls podName:d78a830a-17d2-4940-be08-149b0b789bbb nodeName:}" failed. No retries permitted until 2026-09-16 18:42:14.152665391 +0000 UTC m=+129.340041170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls") pod "image-registry-6dd47fb98c-2v75m" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb") : secret "image-registry-tls" not found
Sep 16 18:42:13.652924 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.652886    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d78a830a-17d2-4940-be08-149b0b789bbb-ca-trust-extracted\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.652986 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.652930    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d78a830a-17d2-4940-be08-149b0b789bbb-trusted-ca\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.653021 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.652993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d78a830a-17d2-4940-be08-149b0b789bbb-registry-certificates\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.654898 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.654874    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d78a830a-17d2-4940-be08-149b0b789bbb-image-registry-private-configuration\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.654992 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.654932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d78a830a-17d2-4940-be08-149b0b789bbb-installation-pull-secrets\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.675166 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.675136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-bound-sa-token\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.675421 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.675394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mgcv\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-kube-api-access-6mgcv\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:13.686394 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.686373    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:13.753388 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.753362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vps\" (UniqueName: \"kubernetes.io/projected/049b76d8-852b-42f0-9720-d8bb5e59ff5f-kube-api-access-q5vps\") pod \"volume-data-source-validator-86695b7cc7-z6q5q\" (UID: \"049b76d8-852b-42f0-9720-d8bb5e59ff5f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z6q5q"
Sep 16 18:42:13.768440 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.768419    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vps\" (UniqueName: \"kubernetes.io/projected/049b76d8-852b-42f0-9720-d8bb5e59ff5f-kube-api-access-q5vps\") pod \"volume-data-source-validator-86695b7cc7-z6q5q\" (UID: \"049b76d8-852b-42f0-9720-d8bb5e59ff5f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z6q5q"
Sep 16 18:42:13.798452 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.798432    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z6q5q"
Sep 16 18:42:13.955546 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:13.955518    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-sjlt4"]
Sep 16 18:42:13.958813 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:13.958780    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aca151a_3031_429f_aa10_fb8d21ba5a27.slice/crio-810cbae61d9c50afd9f686ce68c217b4f59d4134092ae96eafe39b436e6022e0 WatchSource:0}: Error finding container 810cbae61d9c50afd9f686ce68c217b4f59d4134092ae96eafe39b436e6022e0: Status 404 returned error can't find the container with id 810cbae61d9c50afd9f686ce68c217b4f59d4134092ae96eafe39b436e6022e0
Sep 16 18:42:14.071070 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:14.071048    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z6q5q"]
Sep 16 18:42:14.073727 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:14.073703    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049b76d8_852b_42f0_9720_d8bb5e59ff5f.slice/crio-4757f0a85e318da430829c3dc0b5395761c4a4b40ddda3bd963f41dc1dbfd2e4 WatchSource:0}: Error finding container 4757f0a85e318da430829c3dc0b5395761c4a4b40ddda3bd963f41dc1dbfd2e4: Status 404 returned error can't find the container with id 4757f0a85e318da430829c3dc0b5395761c4a4b40ddda3bd963f41dc1dbfd2e4
Sep 16 18:42:14.155957 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:14.155933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:14.156078 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:14.156059    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:42:14.156124 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:14.156080    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd47fb98c-2v75m: secret "image-registry-tls" not found
Sep 16 18:42:14.156157 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:14.156133    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls podName:d78a830a-17d2-4940-be08-149b0b789bbb nodeName:}" failed. No retries permitted until 2026-09-16 18:42:15.156110663 +0000 UTC m=+130.343486428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls") pod "image-registry-6dd47fb98c-2v75m" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb") : secret "image-registry-tls" not found
Sep 16 18:42:14.720621 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:14.720588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z6q5q" event={"ID":"049b76d8-852b-42f0-9720-d8bb5e59ff5f","Type":"ContainerStarted","Data":"4757f0a85e318da430829c3dc0b5395761c4a4b40ddda3bd963f41dc1dbfd2e4"}
Sep 16 18:42:14.722201 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:14.722168    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4" event={"ID":"3aca151a-3031-429f-aa10-fb8d21ba5a27","Type":"ContainerStarted","Data":"810cbae61d9c50afd9f686ce68c217b4f59d4134092ae96eafe39b436e6022e0"}
Sep 16 18:42:15.061605 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:15.061579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:42:15.061722 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:15.061712    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:42:15.061783 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:15.061774    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs podName:47c4bc73-2eff-41ea-91fc-5acce1bcd526 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:17.061753602 +0000 UTC m=+252.249129384 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs") pod "network-metrics-daemon-2bzzg" (UID: "47c4bc73-2eff-41ea-91fc-5acce1bcd526") : secret "metrics-daemon-secret" not found
Sep 16 18:42:15.162515 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:15.162476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:15.162661 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:15.162641    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:42:15.162707 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:15.162667    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd47fb98c-2v75m: secret "image-registry-tls" not found
Sep 16 18:42:15.162741 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:15.162729    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls podName:d78a830a-17d2-4940-be08-149b0b789bbb nodeName:}" failed. No retries permitted until 2026-09-16 18:42:17.162708893 +0000 UTC m=+132.350084674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls") pod "image-registry-6dd47fb98c-2v75m" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb") : secret "image-registry-tls" not found
Sep 16 18:42:15.726777 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:15.726741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z6q5q" event={"ID":"049b76d8-852b-42f0-9720-d8bb5e59ff5f","Type":"ContainerStarted","Data":"ec35dd668733ac9dc5aaa95a570786c79a79a9544baeb7d29db7de4216aef5e0"}
Sep 16 18:42:15.748318 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:15.748186    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z6q5q" podStartSLOduration=1.829209406 podStartE2EDuration="2.748166197s" totalImagesPullingTime="918.956791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:15.748074788 +0000 UTC m=+130.935450576" watchObservedRunningTime="2026-09-16 18:42:15.748166197 +0000 UTC m=+130.935541997"
Sep 16 18:42:16.731435 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:16.731389    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/0.log"
Sep 16 18:42:16.731843 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:16.731454    2582 generic.go:356] "Generic (PLEG): container finished" podID="3aca151a-3031-429f-aa10-fb8d21ba5a27" containerID="35c7302d7ed6e17ba764003c1c7caed7807b40fbe7bf206772e4f7fc0eba686a" exitCode=255
Sep 16 18:42:16.731843 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:16.731564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4" event={"ID":"3aca151a-3031-429f-aa10-fb8d21ba5a27","Type":"ContainerDied","Data":"35c7302d7ed6e17ba764003c1c7caed7807b40fbe7bf206772e4f7fc0eba686a"}
Sep 16 18:42:16.731843 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:16.731761    2582 scope.go:117] "RemoveContainer" containerID="35c7302d7ed6e17ba764003c1c7caed7807b40fbe7bf206772e4f7fc0eba686a"
Sep 16 18:42:17.177046 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:17.177007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:17.177166 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:17.177106    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:42:17.177166 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:17.177116    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd47fb98c-2v75m: secret "image-registry-tls" not found
Sep 16 18:42:17.177166 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:17.177163    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls podName:d78a830a-17d2-4940-be08-149b0b789bbb nodeName:}" failed. No retries permitted until 2026-09-16 18:42:21.177149909 +0000 UTC m=+136.364525674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls") pod "image-registry-6dd47fb98c-2v75m" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb") : secret "image-registry-tls" not found
Sep 16 18:42:17.735886 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:17.735858    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 18:42:17.736317 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:17.736301    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/0.log"
Sep 16 18:42:17.736370 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:17.736339    2582 generic.go:356] "Generic (PLEG): container finished" podID="3aca151a-3031-429f-aa10-fb8d21ba5a27" containerID="c6c3b6a5c5ac670cc0b4b44ded7aa2d9c2a46bff2ae2c6ea6bda77b817d28e14" exitCode=255
Sep 16 18:42:17.736406 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:17.736370    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4" event={"ID":"3aca151a-3031-429f-aa10-fb8d21ba5a27","Type":"ContainerDied","Data":"c6c3b6a5c5ac670cc0b4b44ded7aa2d9c2a46bff2ae2c6ea6bda77b817d28e14"}
Sep 16 18:42:17.736444 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:17.736429    2582 scope.go:117] "RemoveContainer" containerID="35c7302d7ed6e17ba764003c1c7caed7807b40fbe7bf206772e4f7fc0eba686a"
Sep 16 18:42:17.736621 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:17.736602    2582 scope.go:117] "RemoveContainer" containerID="c6c3b6a5c5ac670cc0b4b44ded7aa2d9c2a46bff2ae2c6ea6bda77b817d28e14"
Sep 16 18:42:17.736785 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:17.736762    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-sjlt4_openshift-console-operator(3aca151a-3031-429f-aa10-fb8d21ba5a27)\"" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4" podUID="3aca151a-3031-429f-aa10-fb8d21ba5a27"
Sep 16 18:42:18.264872 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.264843    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p2kqq"]
Sep 16 18:42:18.267798 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.267781    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.270328 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.270306    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:42:18.270852 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.270837    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:42:18.271369 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.271350    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:42:18.271523 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.271435    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:42:18.271620 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.271601    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tvh6w\""
Sep 16 18:42:18.296276 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.296250    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2kqq"]
Sep 16 18:42:18.385885 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.385860    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-crio-socket\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.385984 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.385892    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.385984 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.385972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.386062 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.385990    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lkvb\" (UniqueName: \"kubernetes.io/projected/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-api-access-2lkvb\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.386062 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.386020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-data-volume\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.486555 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.486532    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-crio-socket\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.486620 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.486569    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.486660 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.486630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.486660 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.486650    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-crio-socket\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.486747 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.486659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lkvb\" (UniqueName: \"kubernetes.io/projected/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-api-access-2lkvb\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.486747 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.486698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-data-volume\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.486839 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:18.486749    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:18.486839 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:18.486762    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:18.486839 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:18.486827    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls podName:5bb3ad5a-942a-4f71-a5b4-2c387ca3d376 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:18.98680396 +0000 UTC m=+134.174179787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2kqq" (UID: "5bb3ad5a-942a-4f71-a5b4-2c387ca3d376") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:18.486981 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:18.486842    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm podName:5bb3ad5a-942a-4f71-a5b4-2c387ca3d376 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:18.986835025 +0000 UTC m=+134.174210791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2kqq" (UID: "5bb3ad5a-942a-4f71-a5b4-2c387ca3d376") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:18.487029 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.487013    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-data-volume\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.506807 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.506782    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lkvb\" (UniqueName: \"kubernetes.io/projected/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-api-access-2lkvb\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.741128 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.741102    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 18:42:18.741429 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.741385    2582 scope.go:117] "RemoveContainer" containerID="c6c3b6a5c5ac670cc0b4b44ded7aa2d9c2a46bff2ae2c6ea6bda77b817d28e14"
Sep 16 18:42:18.741542 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:18.741522    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-sjlt4_openshift-console-operator(3aca151a-3031-429f-aa10-fb8d21ba5a27)\"" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4" podUID="3aca151a-3031-429f-aa10-fb8d21ba5a27"
Sep 16 18:42:18.991024 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.990983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.991233 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:18.991070    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:18.991233 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:18.991107    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:18.991233 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:18.991173    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm podName:5bb3ad5a-942a-4f71-a5b4-2c387ca3d376 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:19.991154994 +0000 UTC m=+135.178530759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2kqq" (UID: "5bb3ad5a-942a-4f71-a5b4-2c387ca3d376") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:18.991233 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:18.991175    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:18.991233 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:18.991213    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls podName:5bb3ad5a-942a-4f71-a5b4-2c387ca3d376 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:19.991202208 +0000 UTC m=+135.178577993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2kqq" (UID: "5bb3ad5a-942a-4f71-a5b4-2c387ca3d376") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:19.642432 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:19.642401    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-cs4bd"]
Sep 16 18:42:19.644835 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:19.644817    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cs4bd"
Sep 16 18:42:19.648305 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:19.648269    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:42:19.648305 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:19.648291    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qkk5l\""
Sep 16 18:42:19.648492 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:19.648275    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:42:19.667384 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:19.667360    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-cs4bd"]
Sep 16 18:42:19.798348 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:19.798321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6mbw\" (UniqueName: \"kubernetes.io/projected/0114f015-bf5a-4d44-b6fd-3bb49cc8e372-kube-api-access-w6mbw\") pod \"migrator-8b4d697c5-cs4bd\" (UID: \"0114f015-bf5a-4d44-b6fd-3bb49cc8e372\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cs4bd"
Sep 16 18:42:19.899172 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:19.899117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6mbw\" (UniqueName: \"kubernetes.io/projected/0114f015-bf5a-4d44-b6fd-3bb49cc8e372-kube-api-access-w6mbw\") pod \"migrator-8b4d697c5-cs4bd\" (UID: \"0114f015-bf5a-4d44-b6fd-3bb49cc8e372\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cs4bd"
Sep 16 18:42:19.908795 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:19.908771    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6mbw\" (UniqueName: \"kubernetes.io/projected/0114f015-bf5a-4d44-b6fd-3bb49cc8e372-kube-api-access-w6mbw\") pod \"migrator-8b4d697c5-cs4bd\" (UID: \"0114f015-bf5a-4d44-b6fd-3bb49cc8e372\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cs4bd"
Sep 16 18:42:19.954782 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:19.954758    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cs4bd"
Sep 16 18:42:20.000419 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:20.000390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:20.000548 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:20.000486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:20.000599 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:20.000541    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:20.000644 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:20.000610    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls podName:5bb3ad5a-942a-4f71-a5b4-2c387ca3d376 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:22.000594738 +0000 UTC m=+137.187970515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2kqq" (UID: "5bb3ad5a-942a-4f71-a5b4-2c387ca3d376") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:20.000644 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:20.000617    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:20.000728 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:20.000672    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm podName:5bb3ad5a-942a-4f71-a5b4-2c387ca3d376 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:22.000658527 +0000 UTC m=+137.188034357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2kqq" (UID: "5bb3ad5a-942a-4f71-a5b4-2c387ca3d376") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:20.231126 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:20.231099    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-cs4bd"]
Sep 16 18:42:20.234098 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:20.234071    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0114f015_bf5a_4d44_b6fd_3bb49cc8e372.slice/crio-866076ddc779fe210e4932bdb949ea1e096ce4817327a3bab08b24a09f3e33d5 WatchSource:0}: Error finding container 866076ddc779fe210e4932bdb949ea1e096ce4817327a3bab08b24a09f3e33d5: Status 404 returned error can't find the container with id 866076ddc779fe210e4932bdb949ea1e096ce4817327a3bab08b24a09f3e33d5
Sep 16 18:42:20.344527 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:20.344502    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z5pl8_f8eeb34d-7a85-40ac-8755-be657c455ec6/dns-node-resolver/0.log"
Sep 16 18:42:20.748587 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:20.748555    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cs4bd" event={"ID":"0114f015-bf5a-4d44-b6fd-3bb49cc8e372","Type":"ContainerStarted","Data":"866076ddc779fe210e4932bdb949ea1e096ce4817327a3bab08b24a09f3e33d5"}
Sep 16 18:42:21.209196 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:21.209175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:21.209528 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:21.209312    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:42:21.209528 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:21.209327    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd47fb98c-2v75m: secret "image-registry-tls" not found
Sep 16 18:42:21.209528 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:21.209371    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls podName:d78a830a-17d2-4940-be08-149b0b789bbb nodeName:}" failed. No retries permitted until 2026-09-16 18:42:29.209358346 +0000 UTC m=+144.396734111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls") pod "image-registry-6dd47fb98c-2v75m" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb") : secret "image-registry-tls" not found
Sep 16 18:42:21.349697 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:21.349675    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-shplt_d37b1617-d4af-42a9-8795-2074cbf8be05/node-ca/0.log"
Sep 16 18:42:21.754066 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:21.754033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cs4bd" event={"ID":"0114f015-bf5a-4d44-b6fd-3bb49cc8e372","Type":"ContainerStarted","Data":"86fb86dd21bc59eacb26021c4f077c51b9cfd111f5da614db9cbfbf44bbd721e"}
Sep 16 18:42:21.754208 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:21.754089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cs4bd" event={"ID":"0114f015-bf5a-4d44-b6fd-3bb49cc8e372","Type":"ContainerStarted","Data":"9149e77eee88af83d84e21af6ef5a7e474dc4924841be0df320e2e0517367ef8"}
Sep 16 18:42:21.786115 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:21.786072    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cs4bd" podStartSLOduration=1.8230038240000002 podStartE2EDuration="2.786057091s" totalImagesPullingTime="963.053267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:21.785943285 +0000 UTC m=+136.973319071" watchObservedRunningTime="2026-09-16 18:42:21.786057091 +0000 UTC m=+136.973432877"
Sep 16 18:42:22.013279 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:22.013220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:22.013279 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:22.013277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:22.013456 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:22.013361    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:22.013456 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:22.013375    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:22.013456 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:22.013429    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls podName:5bb3ad5a-942a-4f71-a5b4-2c387ca3d376 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:26.013415913 +0000 UTC m=+141.200791692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2kqq" (UID: "5bb3ad5a-942a-4f71-a5b4-2c387ca3d376") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:22.013456 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:22.013442    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm podName:5bb3ad5a-942a-4f71-a5b4-2c387ca3d376 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:26.013435425 +0000 UTC m=+141.200811190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2kqq" (UID: "5bb3ad5a-942a-4f71-a5b4-2c387ca3d376") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:22.150133 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:22.150071    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xlrfw_a00e8866-0ae0-4245-b716-faac4ee1cd3f/serve-healthcheck-canary/0.log"
Sep 16 18:42:23.686812 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:23.686772    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:23.687235 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:23.686853    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:23.687235 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:23.687166    2582 scope.go:117] "RemoveContainer" containerID="c6c3b6a5c5ac670cc0b4b44ded7aa2d9c2a46bff2ae2c6ea6bda77b817d28e14"
Sep 16 18:42:23.687342 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:23.687320    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-sjlt4_openshift-console-operator(3aca151a-3031-429f-aa10-fb8d21ba5a27)\"" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4" podUID="3aca151a-3031-429f-aa10-fb8d21ba5a27"
Sep 16 18:42:26.042140 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:26.042106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:26.042542 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:26.042164    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:26.042542 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:26.042238    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:26.042542 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:26.042270    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:26.042542 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:26.042300    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm podName:5bb3ad5a-942a-4f71-a5b4-2c387ca3d376 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:34.042285965 +0000 UTC m=+149.229661730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2kqq" (UID: "5bb3ad5a-942a-4f71-a5b4-2c387ca3d376") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:26.042542 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:26.042345    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls podName:5bb3ad5a-942a-4f71-a5b4-2c387ca3d376 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:34.042328384 +0000 UTC m=+149.229704236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2kqq" (UID: "5bb3ad5a-942a-4f71-a5b4-2c387ca3d376") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:29.265987 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:29.265938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:29.269015 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:29.268989    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls\") pod \"image-registry-6dd47fb98c-2v75m\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") " pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:29.390841 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:29.390795    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:29.670855 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:29.670791    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd47fb98c-2v75m"]
Sep 16 18:42:29.674171 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:29.674146    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78a830a_17d2_4940_be08_149b0b789bbb.slice/crio-d465c747fb5a52a95fa6a4a5ce189f754dd9d6db3d92590e0057633fa0453d22 WatchSource:0}: Error finding container d465c747fb5a52a95fa6a4a5ce189f754dd9d6db3d92590e0057633fa0453d22: Status 404 returned error can't find the container with id d465c747fb5a52a95fa6a4a5ce189f754dd9d6db3d92590e0057633fa0453d22
Sep 16 18:42:29.780223 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:29.780193    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m" event={"ID":"d78a830a-17d2-4940-be08-149b0b789bbb","Type":"ContainerStarted","Data":"f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321"}
Sep 16 18:42:29.780326 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:29.780241    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m" event={"ID":"d78a830a-17d2-4940-be08-149b0b789bbb","Type":"ContainerStarted","Data":"d465c747fb5a52a95fa6a4a5ce189f754dd9d6db3d92590e0057633fa0453d22"}
Sep 16 18:42:29.780375 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:29.780323    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:29.806626 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:29.806586    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m" podStartSLOduration=16.806572308 podStartE2EDuration="16.806572308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:29.805974106 +0000 UTC m=+144.993349896" watchObservedRunningTime="2026-09-16 18:42:29.806572308 +0000 UTC m=+144.993948095"
Sep 16 18:42:34.100063 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:34.100021    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:34.100475 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:34.100077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:34.100585 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:34.100562    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:34.102556 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:34.102530    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5bb3ad5a-942a-4f71-a5b4-2c387ca3d376-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2kqq\" (UID: \"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376\") " pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:34.176841 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:34.176811    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2kqq"
Sep 16 18:42:34.453586 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:34.453558    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2kqq"]
Sep 16 18:42:34.456326 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:34.456297    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb3ad5a_942a_4f71_a5b4_2c387ca3d376.slice/crio-908aa568f8a38cb712f5310c5bf60aa1fe5fd6b2fedbdf0f0056208266b8d4ce WatchSource:0}: Error finding container 908aa568f8a38cb712f5310c5bf60aa1fe5fd6b2fedbdf0f0056208266b8d4ce: Status 404 returned error can't find the container with id 908aa568f8a38cb712f5310c5bf60aa1fe5fd6b2fedbdf0f0056208266b8d4ce
Sep 16 18:42:34.800400 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:34.800337    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2kqq" event={"ID":"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376","Type":"ContainerStarted","Data":"15a86e2f151fb5e133084eef19dab420e411aaf962565d48e74a25038b0ed4f8"}
Sep 16 18:42:34.800400 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:34.800394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2kqq" event={"ID":"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376","Type":"ContainerStarted","Data":"908aa568f8a38cb712f5310c5bf60aa1fe5fd6b2fedbdf0f0056208266b8d4ce"}
Sep 16 18:42:35.805426 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:35.805397    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2kqq" event={"ID":"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376","Type":"ContainerStarted","Data":"3dd4364d363add2eb1ce2b15958e75abb83c05b21fae686007a23612c26fe850"}
Sep 16 18:42:36.811659 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:36.811597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2kqq" event={"ID":"5bb3ad5a-942a-4f71-a5b4-2c387ca3d376","Type":"ContainerStarted","Data":"3474f54297da1f9290cc4613d15c7c4e7613e49c5b08d00d1419c30f501a890b"}
Sep 16 18:42:36.835528 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:36.835476    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p2kqq" podStartSLOduration=16.887623361 podStartE2EDuration="18.835459529s" totalImagesPullingTime="1.947836168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:36.834223741 +0000 UTC m=+152.021599527" watchObservedRunningTime="2026-09-16 18:42:36.835459529 +0000 UTC m=+152.022835315"
Sep 16 18:42:39.291289 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:39.291248    2582 scope.go:117] "RemoveContainer" containerID="c6c3b6a5c5ac670cc0b4b44ded7aa2d9c2a46bff2ae2c6ea6bda77b817d28e14"
Sep 16 18:42:39.824365 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:39.824344    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 18:42:39.824496 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:39.824421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4" event={"ID":"3aca151a-3031-429f-aa10-fb8d21ba5a27","Type":"ContainerStarted","Data":"25c468f4ecba3f148e4b8b99a565afe03606f6adb3125a0226744d9c390a4217"}
Sep 16 18:42:39.824717 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:39.824691    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:39.845514 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:39.845477    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4" podStartSLOduration=24.706924545 podStartE2EDuration="26.845464047s" totalImagesPullingTime="2.138539502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:39.844121039 +0000 UTC m=+155.031496827" watchObservedRunningTime="2026-09-16 18:42:39.845464047 +0000 UTC m=+155.032839834"
Sep 16 18:42:39.947555 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:39.947537    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-sjlt4"
Sep 16 18:42:41.039477 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.039442    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww"]
Sep 16 18:42:41.042479 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.042463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww"
Sep 16 18:42:41.052197 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.052180    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pkvl7\""
Sep 16 18:42:41.052197 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.052192    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:42:41.052331 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.052192    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:42:41.057157 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.057132    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww"]
Sep 16 18:42:41.143373 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.143345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4666cfdb-3186-49c5-b678-a49c4c211bcf-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-xsxww\" (UID: \"4666cfdb-3186-49c5-b678-a49c4c211bcf\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww"
Sep 16 18:42:41.143373 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.143371    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4666cfdb-3186-49c5-b678-a49c4c211bcf-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xsxww\" (UID: \"4666cfdb-3186-49c5-b678-a49c4c211bcf\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww"
Sep 16 18:42:41.149380 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.149360    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dd47fb98c-2v75m"]
Sep 16 18:42:41.244390 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.244365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4666cfdb-3186-49c5-b678-a49c4c211bcf-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-xsxww\" (UID: \"4666cfdb-3186-49c5-b678-a49c4c211bcf\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww"
Sep 16 18:42:41.244493 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.244393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4666cfdb-3186-49c5-b678-a49c4c211bcf-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xsxww\" (UID: \"4666cfdb-3186-49c5-b678-a49c4c211bcf\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww"
Sep 16 18:42:41.245042 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.244969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4666cfdb-3186-49c5-b678-a49c4c211bcf-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-xsxww\" (UID: \"4666cfdb-3186-49c5-b678-a49c4c211bcf\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww"
Sep 16 18:42:41.246750 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.246726    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4666cfdb-3186-49c5-b678-a49c4c211bcf-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xsxww\" (UID: \"4666cfdb-3186-49c5-b678-a49c4c211bcf\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww"
Sep 16 18:42:41.351243 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.351196    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww"
Sep 16 18:42:41.624596 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.624568    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww"]
Sep 16 18:42:41.626120 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:41.626096    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4666cfdb_3186_49c5_b678_a49c4c211bcf.slice/crio-1dc0d12ad883713dc4eae6bd5b2c36144eaa6177b126e23253fe83f583855fcb WatchSource:0}: Error finding container 1dc0d12ad883713dc4eae6bd5b2c36144eaa6177b126e23253fe83f583855fcb: Status 404 returned error can't find the container with id 1dc0d12ad883713dc4eae6bd5b2c36144eaa6177b126e23253fe83f583855fcb
Sep 16 18:42:41.832101 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:41.832074    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww" event={"ID":"4666cfdb-3186-49c5-b678-a49c4c211bcf","Type":"ContainerStarted","Data":"1dc0d12ad883713dc4eae6bd5b2c36144eaa6177b126e23253fe83f583855fcb"}
Sep 16 18:42:42.682578 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:42.682549    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hl5jx" podUID="a673e9f7-670e-4604-95f2-7aab6f511727"
Sep 16 18:42:42.836271 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:42.836246    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww" event={"ID":"4666cfdb-3186-49c5-b678-a49c4c211bcf","Type":"ContainerStarted","Data":"5f53faed3ffd15b8f90b8f4800cf005466e7855571b07e87620f443a0a3e7b34"}
Sep 16 18:42:42.836392 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:42.836344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:42:42.862769 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:42.862697    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xsxww" podStartSLOduration=0.856219301 podStartE2EDuration="1.862682153s" totalImagesPullingTime="1.006462852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:42.862257119 +0000 UTC m=+158.049632911" watchObservedRunningTime="2026-09-16 18:42:42.862682153 +0000 UTC m=+158.050057941"
Sep 16 18:42:44.304851 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:44.304805    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2bzzg" podUID="47c4bc73-2eff-41ea-91fc-5acce1bcd526"
Sep 16 18:42:47.587552 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:47.587516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:42:47.590228 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:47.590206    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a673e9f7-670e-4604-95f2-7aab6f511727-metrics-tls\") pod \"dns-default-hl5jx\" (UID: \"a673e9f7-670e-4604-95f2-7aab6f511727\") " pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:42:47.640039 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:47.640010    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xfn4n\""
Sep 16 18:42:47.650031 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:47.650000    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:42:47.927453 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:47.927426    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hl5jx"]
Sep 16 18:42:47.929521 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:47.929498    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda673e9f7_670e_4604_95f2_7aab6f511727.slice/crio-de2fcec21855d6f5cff34a75653a91fb0ed014718722acb89fcd4e522158b3df WatchSource:0}: Error finding container de2fcec21855d6f5cff34a75653a91fb0ed014718722acb89fcd4e522158b3df: Status 404 returned error can't find the container with id de2fcec21855d6f5cff34a75653a91fb0ed014718722acb89fcd4e522158b3df
Sep 16 18:42:48.305508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:48.305441    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj"]
Sep 16 18:42:48.308446 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:48.308425    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj"
Sep 16 18:42:48.311652 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:48.311633    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:42:48.311885 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:48.311850    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nxb5v\""
Sep 16 18:42:48.322464 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:48.322438    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj"]
Sep 16 18:42:48.392925 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:48.392882    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/725304b0-f416-4a92-90ff-d1a3c9be7845-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-jlhdj\" (UID: \"725304b0-f416-4a92-90ff-d1a3c9be7845\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj"
Sep 16 18:42:48.494167 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:48.494143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/725304b0-f416-4a92-90ff-d1a3c9be7845-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-jlhdj\" (UID: \"725304b0-f416-4a92-90ff-d1a3c9be7845\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj"
Sep 16 18:42:48.494296 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:48.494242    2582 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:42:48.494296 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:48.494293    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/725304b0-f416-4a92-90ff-d1a3c9be7845-tls-certificates podName:725304b0-f416-4a92-90ff-d1a3c9be7845 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:48.994277952 +0000 UTC m=+164.181653731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/725304b0-f416-4a92-90ff-d1a3c9be7845-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-jlhdj" (UID: "725304b0-f416-4a92-90ff-d1a3c9be7845") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:42:48.856750 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:48.856708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hl5jx" event={"ID":"a673e9f7-670e-4604-95f2-7aab6f511727","Type":"ContainerStarted","Data":"de2fcec21855d6f5cff34a75653a91fb0ed014718722acb89fcd4e522158b3df"}
Sep 16 18:42:48.998487 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:48.998458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/725304b0-f416-4a92-90ff-d1a3c9be7845-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-jlhdj\" (UID: \"725304b0-f416-4a92-90ff-d1a3c9be7845\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj"
Sep 16 18:42:49.001107 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:49.001077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/725304b0-f416-4a92-90ff-d1a3c9be7845-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-jlhdj\" (UID: \"725304b0-f416-4a92-90ff-d1a3c9be7845\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj"
Sep 16 18:42:49.219411 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:49.219378    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj"
Sep 16 18:42:49.604534 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:49.604509    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj"]
Sep 16 18:42:49.607492 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:49.607469    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725304b0_f416_4a92_90ff_d1a3c9be7845.slice/crio-e1f004e18aef4f4da3fe2eeda97c4e52e7428873e33211044582569463e7caf1 WatchSource:0}: Error finding container e1f004e18aef4f4da3fe2eeda97c4e52e7428873e33211044582569463e7caf1: Status 404 returned error can't find the container with id e1f004e18aef4f4da3fe2eeda97c4e52e7428873e33211044582569463e7caf1
Sep 16 18:42:49.860602 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:49.860543    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj" event={"ID":"725304b0-f416-4a92-90ff-d1a3c9be7845","Type":"ContainerStarted","Data":"e1f004e18aef4f4da3fe2eeda97c4e52e7428873e33211044582569463e7caf1"}
Sep 16 18:42:49.862316 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:49.862293    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hl5jx" event={"ID":"a673e9f7-670e-4604-95f2-7aab6f511727","Type":"ContainerStarted","Data":"6dde618d48ecf673fc3aded72c0846b0ffd3d691f0e9fe63e18b60eb9bcaef4c"}
Sep 16 18:42:49.862400 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:49.862339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hl5jx" event={"ID":"a673e9f7-670e-4604-95f2-7aab6f511727","Type":"ContainerStarted","Data":"c87e103f438a2b1ed94f53eb5278e6a7832570b0a7683873ef46b05b708e74fb"}
Sep 16 18:42:49.862447 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:49.862434    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:42:49.884540 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:49.884501    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hl5jx" podStartSLOduration=129.441501769 podStartE2EDuration="2m10.884489648s" totalImagesPullingTime="1.442987879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:49.883165836 +0000 UTC m=+165.070541624" watchObservedRunningTime="2026-09-16 18:42:49.884489648 +0000 UTC m=+165.071865448"
Sep 16 18:42:50.867209 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:50.867125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj" event={"ID":"725304b0-f416-4a92-90ff-d1a3c9be7845","Type":"ContainerStarted","Data":"0c6baf09698fd7db5c00e24c4737543ee772534c0f3d9586893656a404a5b822"}
Sep 16 18:42:50.867622 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:50.867260    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj"
Sep 16 18:42:50.871777 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:50.871758    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj"
Sep 16 18:42:50.890600 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:50.890564    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-jlhdj" podStartSLOduration=1.958585207 podStartE2EDuration="2.89055141s" totalImagesPullingTime="931.966203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:50.890102772 +0000 UTC m=+166.077478576" watchObservedRunningTime="2026-09-16 18:42:50.89055141 +0000 UTC m=+166.077927196"
Sep 16 18:42:51.154410 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.154348    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:42:51.372510 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.372485    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"]
Sep 16 18:42:51.375510 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.375495    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:51.378048 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.378028    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:42:51.378161 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.378081    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:42:51.378161 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.378088    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:42:51.378161 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.378151    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:42:51.378484 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.378471    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:42:51.378702 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.378690    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ms4zs\""
Sep 16 18:42:51.387628 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.387598    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"]
Sep 16 18:42:51.414770 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.414710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e448bd5-002d-4903-9c60-6f5077f88bd5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:51.414862 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.414790    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e448bd5-002d-4903-9c60-6f5077f88bd5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:51.414862 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.414815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e448bd5-002d-4903-9c60-6f5077f88bd5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:51.414862 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.414835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhvh\" (UniqueName: \"kubernetes.io/projected/4e448bd5-002d-4903-9c60-6f5077f88bd5-kube-api-access-wlhvh\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:51.515357 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.515322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e448bd5-002d-4903-9c60-6f5077f88bd5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:51.515464 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.515375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e448bd5-002d-4903-9c60-6f5077f88bd5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:51.515464 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.515394    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e448bd5-002d-4903-9c60-6f5077f88bd5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:51.515464 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.515411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhvh\" (UniqueName: \"kubernetes.io/projected/4e448bd5-002d-4903-9c60-6f5077f88bd5-kube-api-access-wlhvh\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:51.515580 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:51.515498    2582 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 18:42:51.515580 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:51.515565    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e448bd5-002d-4903-9c60-6f5077f88bd5-prometheus-operator-tls podName:4e448bd5-002d-4903-9c60-6f5077f88bd5 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:52.015547223 +0000 UTC m=+167.202922990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/4e448bd5-002d-4903-9c60-6f5077f88bd5-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-bfqmj" (UID: "4e448bd5-002d-4903-9c60-6f5077f88bd5") : secret "prometheus-operator-tls" not found
Sep 16 18:42:51.516089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.516073    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e448bd5-002d-4903-9c60-6f5077f88bd5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:51.517970 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.517948    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e448bd5-002d-4903-9c60-6f5077f88bd5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:51.525603 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:51.525581    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhvh\" (UniqueName: \"kubernetes.io/projected/4e448bd5-002d-4903-9c60-6f5077f88bd5-kube-api-access-wlhvh\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:52.018249 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:52.018216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e448bd5-002d-4903-9c60-6f5077f88bd5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:52.020886 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:52.020864    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e448bd5-002d-4903-9c60-6f5077f88bd5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-bfqmj\" (UID: \"4e448bd5-002d-4903-9c60-6f5077f88bd5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:52.286038 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:52.285971    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"
Sep 16 18:42:52.562318 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:52.562259    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj"]
Sep 16 18:42:52.564996 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:52.564968    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e448bd5_002d_4903_9c60_6f5077f88bd5.slice/crio-d608b9919fc297736ecc9327354324e96e0f956327dc787762bd95cae34c6bf7 WatchSource:0}: Error finding container d608b9919fc297736ecc9327354324e96e0f956327dc787762bd95cae34c6bf7: Status 404 returned error can't find the container with id d608b9919fc297736ecc9327354324e96e0f956327dc787762bd95cae34c6bf7
Sep 16 18:42:52.874690 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:52.874619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj" event={"ID":"4e448bd5-002d-4903-9c60-6f5077f88bd5","Type":"ContainerStarted","Data":"d608b9919fc297736ecc9327354324e96e0f956327dc787762bd95cae34c6bf7"}
Sep 16 18:42:53.880580 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:53.880552    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj" event={"ID":"4e448bd5-002d-4903-9c60-6f5077f88bd5","Type":"ContainerStarted","Data":"442554b5bedbc85da9a818ab655b31d3765ef4bb82c07e75e80d818de0a0bf76"}
Sep 16 18:42:53.880981 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:53.880610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj" event={"ID":"4e448bd5-002d-4903-9c60-6f5077f88bd5","Type":"ContainerStarted","Data":"91c2c8b5e4417c34ef8b46c363b6a6cb6daae9993ed96db2bfb3f1ae3b5ca7c2"}
Sep 16 18:42:53.900944 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:53.900856    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bfqmj" podStartSLOduration=1.736633097 podStartE2EDuration="2.900837468s" totalImagesPullingTime="1.164204371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:53.90013157 +0000 UTC m=+169.087507357" watchObservedRunningTime="2026-09-16 18:42:53.900837468 +0000 UTC m=+169.088213257"
Sep 16 18:42:55.338411 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.338386    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fdf7f869b-rk5s8"]
Sep 16 18:42:55.343692 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.343672    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.346981 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.346956    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:42:55.347197 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.347182    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:42:55.347872 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.347852    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:42:55.347872 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.347865    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:42:55.348063 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.347859    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:42:55.348063 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.347865    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:42:55.348063 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.347859    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:42:55.348063 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.347894    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ph8qj\""
Sep 16 18:42:55.358976 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.358956    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fdf7f869b-rk5s8"]
Sep 16 18:42:55.443745 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.443714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-service-ca\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.443869 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.443758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36658226-bed8-4eef-b166-2a8b2425346e-console-oauth-config\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.443869 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.443814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-oauth-serving-cert\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.443869 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.443862    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36658226-bed8-4eef-b166-2a8b2425346e-console-serving-cert\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.444063 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.443888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zxvw\" (UniqueName: \"kubernetes.io/projected/36658226-bed8-4eef-b166-2a8b2425346e-kube-api-access-2zxvw\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.444063 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.443959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-console-config\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.544713 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.544686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36658226-bed8-4eef-b166-2a8b2425346e-console-serving-cert\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.544713 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.544713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zxvw\" (UniqueName: \"kubernetes.io/projected/36658226-bed8-4eef-b166-2a8b2425346e-kube-api-access-2zxvw\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.544849 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.544729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-console-config\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.544949 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.544929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-service-ca\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.544990 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.544963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36658226-bed8-4eef-b166-2a8b2425346e-console-oauth-config\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.545038 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.544988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-oauth-serving-cert\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.545442 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.545418    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-console-config\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.545651 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.545630    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-oauth-serving-cert\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.545699 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.545630    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-service-ca\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.547189 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.547171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36658226-bed8-4eef-b166-2a8b2425346e-console-serving-cert\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.547339 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.547323    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36658226-bed8-4eef-b166-2a8b2425346e-console-oauth-config\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.558629 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.558601    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zxvw\" (UniqueName: \"kubernetes.io/projected/36658226-bed8-4eef-b166-2a8b2425346e-kube-api-access-2zxvw\") pod \"console-7fdf7f869b-rk5s8\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") " pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.653536 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.653479    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:42:55.878325 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.878292    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"]
Sep 16 18:42:55.880397 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.880382    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:55.882954 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.882930    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-72hvd\""
Sep 16 18:42:55.883413 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.883394    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:42:55.884088 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.884070    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:42:55.884168 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.884070    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:42:55.894398 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.894377    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"]
Sep 16 18:42:55.922450 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.922401    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w26ms"]
Sep 16 18:42:55.924622 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.924609    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:55.928218 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.928204    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:42:55.928742 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.928705    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:42:55.929064 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.929022    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6dgwx\""
Sep 16 18:42:55.929064 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.929052    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:42:55.943074 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.943055    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fdf7f869b-rk5s8"]
Sep 16 18:42:55.945441 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:55.945422    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36658226_bed8_4eef_b166_2a8b2425346e.slice/crio-0ae5457adb92dd200fa918a2c31194d507c2f36fc10a86088b26464192749ffa WatchSource:0}: Error finding container 0ae5457adb92dd200fa918a2c31194d507c2f36fc10a86088b26464192749ffa: Status 404 returned error can't find the container with id 0ae5457adb92dd200fa918a2c31194d507c2f36fc10a86088b26464192749ffa
Sep 16 18:42:55.947465 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acd962b5-4f89-4adc-921f-807869ec5abd-sys\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:55.947552 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:55.947552 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:55.947552 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947512    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/02eb8316-86a6-4e1b-80b0-5554a341a8aa-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:55.947552 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qns75\" (UniqueName: \"kubernetes.io/projected/acd962b5-4f89-4adc-921f-807869ec5abd-kube-api-access-qns75\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:55.947698 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02eb8316-86a6-4e1b-80b0-5554a341a8aa-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:55.947698 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947646    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-wtmp\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:55.947698 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5w5s\" (UniqueName: \"kubernetes.io/projected/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-api-access-b5w5s\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:55.947811 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-accelerators-collector-config\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:55.947811 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:55.947811 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acd962b5-4f89-4adc-921f-807869ec5abd-metrics-client-ca\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:55.948000 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947837    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/acd962b5-4f89-4adc-921f-807869ec5abd-root\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:55.948000 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947867    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-tls\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:55.948000 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:55.948144 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:55.947999    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-textfile\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.049451 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-wtmp\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.049550 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5w5s\" (UniqueName: \"kubernetes.io/projected/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-api-access-b5w5s\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.049550 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-accelerators-collector-config\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.049550 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.049710 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acd962b5-4f89-4adc-921f-807869ec5abd-metrics-client-ca\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.049710 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-wtmp\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.049710 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/acd962b5-4f89-4adc-921f-807869ec5abd-root\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.050056 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-tls\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.050056 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.050056 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/acd962b5-4f89-4adc-921f-807869ec5abd-root\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.050056 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049818    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-textfile\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.050056 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:56.049843    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:42:56.050056 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acd962b5-4f89-4adc-921f-807869ec5abd-sys\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.050056 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:56.049897    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-tls podName:acd962b5-4f89-4adc-921f-807869ec5abd nodeName:}" failed. No retries permitted until 2026-09-16 18:42:56.549879616 +0000 UTC m=+171.737255386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-tls") pod "node-exporter-w26ms" (UID: "acd962b5-4f89-4adc-921f-807869ec5abd") : secret "node-exporter-tls" not found
Sep 16 18:42:56.050056 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.050056 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.050056 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.050011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/02eb8316-86a6-4e1b-80b0-5554a341a8aa-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.050056 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.050036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qns75\" (UniqueName: \"kubernetes.io/projected/acd962b5-4f89-4adc-921f-807869ec5abd-kube-api-access-qns75\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.050576 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.050069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02eb8316-86a6-4e1b-80b0-5554a341a8aa-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.050576 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.050111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-accelerators-collector-config\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.050576 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:56.050192    2582 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 16 18:42:56.050576 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.050210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-textfile\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.050576 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.050230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.050576 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:42:56.050242    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-state-metrics-tls podName:02eb8316-86a6-4e1b-80b0-5554a341a8aa nodeName:}" failed. No retries permitted until 2026-09-16 18:42:56.55022674 +0000 UTC m=+171.737602524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-qzjqh" (UID: "02eb8316-86a6-4e1b-80b0-5554a341a8aa") : secret "kube-state-metrics-tls" not found
Sep 16 18:42:56.050576 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.049947    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acd962b5-4f89-4adc-921f-807869ec5abd-sys\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.050576 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.050306    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acd962b5-4f89-4adc-921f-807869ec5abd-metrics-client-ca\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.050576 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.050449    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/02eb8316-86a6-4e1b-80b0-5554a341a8aa-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.050881 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.050674    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02eb8316-86a6-4e1b-80b0-5554a341a8aa-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.052293 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.052270    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.052582 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.052563    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.057494 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.057478    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5w5s\" (UniqueName: \"kubernetes.io/projected/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-api-access-b5w5s\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.058953 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.058936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qns75\" (UniqueName: \"kubernetes.io/projected/acd962b5-4f89-4adc-921f-807869ec5abd-kube-api-access-qns75\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.554630 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.554554    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-tls\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.554630 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.554607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.557370 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.557342    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/acd962b5-4f89-4adc-921f-807869ec5abd-node-exporter-tls\") pod \"node-exporter-w26ms\" (UID: \"acd962b5-4f89-4adc-921f-807869ec5abd\") " pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.557556 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.557538    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02eb8316-86a6-4e1b-80b0-5554a341a8aa-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-qzjqh\" (UID: \"02eb8316-86a6-4e1b-80b0-5554a341a8aa\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.789660 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.789631    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"
Sep 16 18:42:56.833030 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.832928    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w26ms"
Sep 16 18:42:56.845573 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:56.845508    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd962b5_4f89_4adc_921f_807869ec5abd.slice/crio-eff383abf1d75e94f1ddebb73dd740f817a729f0b4398a34aa17e6b60cbd68df WatchSource:0}: Error finding container eff383abf1d75e94f1ddebb73dd740f817a729f0b4398a34aa17e6b60cbd68df: Status 404 returned error can't find the container with id eff383abf1d75e94f1ddebb73dd740f817a729f0b4398a34aa17e6b60cbd68df
Sep 16 18:42:56.892843 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.892812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fdf7f869b-rk5s8" event={"ID":"36658226-bed8-4eef-b166-2a8b2425346e","Type":"ContainerStarted","Data":"0ae5457adb92dd200fa918a2c31194d507c2f36fc10a86088b26464192749ffa"}
Sep 16 18:42:56.894250 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.894221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w26ms" event={"ID":"acd962b5-4f89-4adc-921f-807869ec5abd","Type":"ContainerStarted","Data":"eff383abf1d75e94f1ddebb73dd740f817a729f0b4398a34aa17e6b60cbd68df"}
Sep 16 18:42:56.988704 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.988675    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:56.992129 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.992093    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:56.996009 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.995989    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:42:56.996430 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.996406    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:42:56.996528 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.996454    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:42:56.996591 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.996528    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:42:56.996788 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.996769    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:42:56.996976 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.996866    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:42:56.997099 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.997006    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dcnxc\""
Sep 16 18:42:56.997099 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.997096    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:42:56.997199 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:56.997129    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:42:57.006289 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.006271    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:42:57.014707 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.014688    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:57.058785 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.058750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c578544b-7ad9-4218-bd38-0394eeb17165-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.058924 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.058786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c578544b-7ad9-4218-bd38-0394eeb17165-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.058924 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.058821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.058924 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.058885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.059096 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.058959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c578544b-7ad9-4218-bd38-0394eeb17165-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.059096 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.058976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.059096 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.059004    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-config-volume\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.059096 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.059046    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.059096 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.059071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg6x6\" (UniqueName: \"kubernetes.io/projected/c578544b-7ad9-4218-bd38-0394eeb17165-kube-api-access-zg6x6\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.059096 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.059088    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-web-config\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.059382 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.059125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.059382 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.059201    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c578544b-7ad9-4218-bd38-0394eeb17165-config-out\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.059382 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.059252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c578544b-7ad9-4218-bd38-0394eeb17165-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.086991 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.086922    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh"]
Sep 16 18:42:57.090068 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:57.090041    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02eb8316_86a6_4e1b_80b0_5554a341a8aa.slice/crio-c857f76fd7b84c770fde13bc39eacd364fc865b38f14a0f7e9ffbd03f8ba4eae WatchSource:0}: Error finding container c857f76fd7b84c770fde13bc39eacd364fc865b38f14a0f7e9ffbd03f8ba4eae: Status 404 returned error can't find the container with id c857f76fd7b84c770fde13bc39eacd364fc865b38f14a0f7e9ffbd03f8ba4eae
Sep 16 18:42:57.160397 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.160371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c578544b-7ad9-4218-bd38-0394eeb17165-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.160531 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.160426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c578544b-7ad9-4218-bd38-0394eeb17165-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.160531 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.160451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c578544b-7ad9-4218-bd38-0394eeb17165-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.160531 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.160516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.160690 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.160543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.160690 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.160613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c578544b-7ad9-4218-bd38-0394eeb17165-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.160690 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.160635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.160857 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.160836    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c578544b-7ad9-4218-bd38-0394eeb17165-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.161039 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.161016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-config-volume\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.161104 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.161072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.161170 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.161103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg6x6\" (UniqueName: \"kubernetes.io/projected/c578544b-7ad9-4218-bd38-0394eeb17165-kube-api-access-zg6x6\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.161170 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.161130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-web-config\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.161269 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.161168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.161269 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.161211    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c578544b-7ad9-4218-bd38-0394eeb17165-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.161819 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.161517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c578544b-7ad9-4218-bd38-0394eeb17165-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.162181 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.162096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c578544b-7ad9-4218-bd38-0394eeb17165-config-out\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.163950 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.163897    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.164187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.164142    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c578544b-7ad9-4218-bd38-0394eeb17165-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.164801 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.164782    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-config-volume\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.164953 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.164934    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.165632 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.165519    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.165632 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.165621    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.165820 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.165802    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-web-config\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.166457 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.166428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.166627 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.166594    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c578544b-7ad9-4218-bd38-0394eeb17165-config-out\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.171264 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.171243    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg6x6\" (UniqueName: \"kubernetes.io/projected/c578544b-7ad9-4218-bd38-0394eeb17165-kube-api-access-zg6x6\") pod \"alertmanager-main-0\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.304004 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.303975    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:57.611509 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.611480    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:57.899839 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:57.899750    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh" event={"ID":"02eb8316-86a6-4e1b-80b0-5554a341a8aa","Type":"ContainerStarted","Data":"c857f76fd7b84c770fde13bc39eacd364fc865b38f14a0f7e9ffbd03f8ba4eae"}
Sep 16 18:42:58.291497 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:58.291461    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:42:58.616703 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:58.616635    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc578544b_7ad9_4218_bd38_0394eeb17165.slice/crio-df8ae7a620882ce4689d75a65b68e98f1918a28fed39f35916c1d9b355c00abe WatchSource:0}: Error finding container df8ae7a620882ce4689d75a65b68e98f1918a28fed39f35916c1d9b355c00abe: Status 404 returned error can't find the container with id df8ae7a620882ce4689d75a65b68e98f1918a28fed39f35916c1d9b355c00abe
Sep 16 18:42:58.905377 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:58.905260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerStarted","Data":"df8ae7a620882ce4689d75a65b68e98f1918a28fed39f35916c1d9b355c00abe"}
Sep 16 18:42:59.125086 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.125059    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c5f686858-5wp7s"]
Sep 16 18:42:59.128263 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.128241    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.133656 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.133635    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:42:59.133766 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.133748    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:42:59.134211 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.134196    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:42:59.134662 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.134638    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:42:59.134784 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.134757    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-44ifmbje9if3q\""
Sep 16 18:42:59.135060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.135033    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-msqxb\""
Sep 16 18:42:59.135709 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.135689    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:42:59.168631 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.168577    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c5f686858-5wp7s"]
Sep 16 18:42:59.179653 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.179617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-tls\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.179767 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.179684    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.179767 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.179714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmvv6\" (UniqueName: \"kubernetes.io/projected/65f47594-f4f8-4a74-908b-1f7810167c04-kube-api-access-qmvv6\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.179767 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.179748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.180042 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.179805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.180042 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.179836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.180042 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.179929    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65f47594-f4f8-4a74-908b-1f7810167c04-metrics-client-ca\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.180042 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.179963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-grpc-tls\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.281441 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.281410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.281441 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.281455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmvv6\" (UniqueName: \"kubernetes.io/projected/65f47594-f4f8-4a74-908b-1f7810167c04-kube-api-access-qmvv6\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.281681 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.281491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.281681 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.281523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.281681 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.281549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.281681 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.281600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65f47594-f4f8-4a74-908b-1f7810167c04-metrics-client-ca\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.281681 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.281632    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-grpc-tls\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.281681 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.281683    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-tls\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.284061 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.284016    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65f47594-f4f8-4a74-908b-1f7810167c04-metrics-client-ca\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.286992 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.286943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-grpc-tls\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.287520 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.287473    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.288237 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.288194    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-tls\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.288560 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.288525    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.289519 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.289467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.290796 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.290774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/65f47594-f4f8-4a74-908b-1f7810167c04-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.297635 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.297589    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmvv6\" (UniqueName: \"kubernetes.io/projected/65f47594-f4f8-4a74-908b-1f7810167c04-kube-api-access-qmvv6\") pod \"thanos-querier-7c5f686858-5wp7s\" (UID: \"65f47594-f4f8-4a74-908b-1f7810167c04\") " pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.438798 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.438777    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:42:59.751295 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.751212    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c5f686858-5wp7s"]
Sep 16 18:42:59.813630 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:42:59.813598    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f47594_f4f8_4a74_908b_1f7810167c04.slice/crio-9dfa76a1145204b0c1fd3bc04a19c787cb3a2fea4d8cb41950942f554618eb8e WatchSource:0}: Error finding container 9dfa76a1145204b0c1fd3bc04a19c787cb3a2fea4d8cb41950942f554618eb8e: Status 404 returned error can't find the container with id 9dfa76a1145204b0c1fd3bc04a19c787cb3a2fea4d8cb41950942f554618eb8e
Sep 16 18:42:59.870163 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.870133    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hl5jx"
Sep 16 18:42:59.912050 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.912020    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh" event={"ID":"02eb8316-86a6-4e1b-80b0-5554a341a8aa","Type":"ContainerStarted","Data":"f545fbc8ed4ae78c41a614c7082138109d47d288b6b4e43d1fd8ecca97e75434"}
Sep 16 18:42:59.912170 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.912088    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh" event={"ID":"02eb8316-86a6-4e1b-80b0-5554a341a8aa","Type":"ContainerStarted","Data":"b84f9c68a4c589b8510739e5afdb0398fe717d1068dd9bc3667167accd8d08da"}
Sep 16 18:42:59.912170 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.912118    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh" event={"ID":"02eb8316-86a6-4e1b-80b0-5554a341a8aa","Type":"ContainerStarted","Data":"5f71991204e5f807cfc3403fa74292003c3883122f821bba1c475b10c5ba78fa"}
Sep 16 18:42:59.913515 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.913496    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s" event={"ID":"65f47594-f4f8-4a74-908b-1f7810167c04","Type":"ContainerStarted","Data":"9dfa76a1145204b0c1fd3bc04a19c787cb3a2fea4d8cb41950942f554618eb8e"}
Sep 16 18:42:59.915378 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.915356    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerStarted","Data":"47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557"}
Sep 16 18:42:59.917501 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.917475    2582 generic.go:356] "Generic (PLEG): container finished" podID="acd962b5-4f89-4adc-921f-807869ec5abd" containerID="19af972402d9d5dda61715d022ad7e04017712dc914fa86425bb84b9090cf37f" exitCode=0
Sep 16 18:42:59.917613 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.917579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w26ms" event={"ID":"acd962b5-4f89-4adc-921f-807869ec5abd","Type":"ContainerDied","Data":"19af972402d9d5dda61715d022ad7e04017712dc914fa86425bb84b9090cf37f"}
Sep 16 18:42:59.931427 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.931395    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fdf7f869b-rk5s8" event={"ID":"36658226-bed8-4eef-b166-2a8b2425346e","Type":"ContainerStarted","Data":"98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82"}
Sep 16 18:42:59.952765 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:42:59.952334    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qzjqh" podStartSLOduration=2.881897822 podStartE2EDuration="4.952318394s" totalImagesPullingTime="2.070420572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:59.951664573 +0000 UTC m=+175.139040361" watchObservedRunningTime="2026-09-16 18:42:59.952318394 +0000 UTC m=+175.139694182"
Sep 16 18:43:00.034977 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.034935    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fdf7f869b-rk5s8" podStartSLOduration=1.820131445 podStartE2EDuration="5.034898773s" totalImagesPullingTime="3.214767328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:00.033791454 +0000 UTC m=+175.221167242" watchObservedRunningTime="2026-09-16 18:43:00.034898773 +0000 UTC m=+175.222274559"
Sep 16 18:43:00.570688 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.570643    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-644fdd7df9-k8t76"]
Sep 16 18:43:00.573609 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.573584    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.576767 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.576612    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:43:00.576767 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.576629    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:43:00.576767 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.576612    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:43:00.576767 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.576613    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kp8jc\""
Sep 16 18:43:00.577169 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.577147    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ft2ppe08uvptf\""
Sep 16 18:43:00.577338 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.577305    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:43:00.584378 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.584357    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"]
Sep 16 18:43:00.586472 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.586457    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"
Sep 16 18:43:00.589768 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.589721    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-644fdd7df9-k8t76"]
Sep 16 18:43:00.593513 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.593197    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:43:00.593513 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.593457    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xhtjb\""
Sep 16 18:43:00.612408 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.612386    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"]
Sep 16 18:43:00.695387 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.695362    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6638b647-14f2-4098-8ce9-06175618d722-client-ca-bundle\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.695506 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.695395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6638b647-14f2-4098-8ce9-06175618d722-metrics-server-audit-profiles\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.695506 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.695436    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff866591-f0fb-4b72-b107-92009c38c858-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-9gpnh\" (UID: \"ff866591-f0fb-4b72-b107-92009c38c858\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"
Sep 16 18:43:00.695613 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.695539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6638b647-14f2-4098-8ce9-06175618d722-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.695613 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.695582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g4d6\" (UniqueName: \"kubernetes.io/projected/ff866591-f0fb-4b72-b107-92009c38c858-kube-api-access-8g4d6\") pod \"monitoring-plugin-5d85778b9d-9gpnh\" (UID: \"ff866591-f0fb-4b72-b107-92009c38c858\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"
Sep 16 18:43:00.695613 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.695608    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ldqs\" (UniqueName: \"kubernetes.io/projected/6638b647-14f2-4098-8ce9-06175618d722-kube-api-access-4ldqs\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.695750 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.695638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6638b647-14f2-4098-8ce9-06175618d722-secret-metrics-server-client-certs\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.695750 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.695731    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6638b647-14f2-4098-8ce9-06175618d722-audit-log\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.695840 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.695766    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6638b647-14f2-4098-8ce9-06175618d722-secret-metrics-server-tls\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.796284 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.796255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6638b647-14f2-4098-8ce9-06175618d722-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.796720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.796302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g4d6\" (UniqueName: \"kubernetes.io/projected/ff866591-f0fb-4b72-b107-92009c38c858-kube-api-access-8g4d6\") pod \"monitoring-plugin-5d85778b9d-9gpnh\" (UID: \"ff866591-f0fb-4b72-b107-92009c38c858\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"
Sep 16 18:43:00.796720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.796328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ldqs\" (UniqueName: \"kubernetes.io/projected/6638b647-14f2-4098-8ce9-06175618d722-kube-api-access-4ldqs\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.796720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.796362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6638b647-14f2-4098-8ce9-06175618d722-secret-metrics-server-client-certs\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.796720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.796422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6638b647-14f2-4098-8ce9-06175618d722-audit-log\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.796720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.796450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6638b647-14f2-4098-8ce9-06175618d722-secret-metrics-server-tls\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.796720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.796574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6638b647-14f2-4098-8ce9-06175618d722-client-ca-bundle\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.796720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.796611    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6638b647-14f2-4098-8ce9-06175618d722-metrics-server-audit-profiles\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.796720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.796650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff866591-f0fb-4b72-b107-92009c38c858-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-9gpnh\" (UID: \"ff866591-f0fb-4b72-b107-92009c38c858\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"
Sep 16 18:43:00.797170 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.796858    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6638b647-14f2-4098-8ce9-06175618d722-audit-log\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.797170 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.797089    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6638b647-14f2-4098-8ce9-06175618d722-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.798241 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.798216    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6638b647-14f2-4098-8ce9-06175618d722-metrics-server-audit-profiles\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.799486 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.799465    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6638b647-14f2-4098-8ce9-06175618d722-secret-metrics-server-client-certs\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.799682 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.799662    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6638b647-14f2-4098-8ce9-06175618d722-secret-metrics-server-tls\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.799797 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.799781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff866591-f0fb-4b72-b107-92009c38c858-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-9gpnh\" (UID: \"ff866591-f0fb-4b72-b107-92009c38c858\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"
Sep 16 18:43:00.799955 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.799935    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6638b647-14f2-4098-8ce9-06175618d722-client-ca-bundle\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.811890 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.811870    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g4d6\" (UniqueName: \"kubernetes.io/projected/ff866591-f0fb-4b72-b107-92009c38c858-kube-api-access-8g4d6\") pod \"monitoring-plugin-5d85778b9d-9gpnh\" (UID: \"ff866591-f0fb-4b72-b107-92009c38c858\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"
Sep 16 18:43:00.812418 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.812395    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ldqs\" (UniqueName: \"kubernetes.io/projected/6638b647-14f2-4098-8ce9-06175618d722-kube-api-access-4ldqs\") pod \"metrics-server-644fdd7df9-k8t76\" (UID: \"6638b647-14f2-4098-8ce9-06175618d722\") " pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.885117 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.885071    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:00.900159 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.899983    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"
Sep 16 18:43:00.938392 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.938308    2582 generic.go:356] "Generic (PLEG): container finished" podID="c578544b-7ad9-4218-bd38-0394eeb17165" containerID="47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557" exitCode=0
Sep 16 18:43:00.938561 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.938411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerDied","Data":"47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557"}
Sep 16 18:43:00.947770 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.946821    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w26ms" event={"ID":"acd962b5-4f89-4adc-921f-807869ec5abd","Type":"ContainerStarted","Data":"b0fe0692f3007def258bd65d861221e7f982361a06013c50777e27c31d9c56a2"}
Sep 16 18:43:00.947770 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:00.946869    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w26ms" event={"ID":"acd962b5-4f89-4adc-921f-807869ec5abd","Type":"ContainerStarted","Data":"00d0283074bab3e3729d19be55d2054960aba44e9f21880dae06ef3da7770d20"}
Sep 16 18:43:01.016150 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:01.016081    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w26ms" podStartSLOduration=3.5881290310000002 podStartE2EDuration="6.016060269s" totalImagesPullingTime="2.315426701s" totalInitContainerRuntime="112.504537ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:01.014776162 +0000 UTC m=+176.202151937" watchObservedRunningTime="2026-09-16 18:43:01.016060269 +0000 UTC m=+176.203436057"
Sep 16 18:43:01.185025 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:01.184900    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-644fdd7df9-k8t76"]
Sep 16 18:43:01.186749 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:01.186726    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"]
Sep 16 18:43:01.491748 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:43:01.491622    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6638b647_14f2_4098_8ce9_06175618d722.slice/crio-d67d3231f3985e9662600884bb4e6612c2017c9d8eadfd54aeb48a3f05597241 WatchSource:0}: Error finding container d67d3231f3985e9662600884bb4e6612c2017c9d8eadfd54aeb48a3f05597241: Status 404 returned error can't find the container with id d67d3231f3985e9662600884bb4e6612c2017c9d8eadfd54aeb48a3f05597241
Sep 16 18:43:01.492657 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:43:01.492634    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff866591_f0fb_4b72_b107_92009c38c858.slice/crio-d06b77412271c7b3bc5ff4551dfcd2633dae04c1c2d0a13601d190da50fe1aed WatchSource:0}: Error finding container d06b77412271c7b3bc5ff4551dfcd2633dae04c1c2d0a13601d190da50fe1aed: Status 404 returned error can't find the container with id d06b77412271c7b3bc5ff4551dfcd2633dae04c1c2d0a13601d190da50fe1aed
Sep 16 18:43:01.954352 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:01.954283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh" event={"ID":"ff866591-f0fb-4b72-b107-92009c38c858","Type":"ContainerStarted","Data":"d06b77412271c7b3bc5ff4551dfcd2633dae04c1c2d0a13601d190da50fe1aed"}
Sep 16 18:43:01.961615 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:01.961433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s" event={"ID":"65f47594-f4f8-4a74-908b-1f7810167c04","Type":"ContainerStarted","Data":"23007b9288b733955816ca25db03016d6076079e75c3f7cbbab30d0aa16a035f"}
Sep 16 18:43:01.961615 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:01.961511    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s" event={"ID":"65f47594-f4f8-4a74-908b-1f7810167c04","Type":"ContainerStarted","Data":"575d011484a82b453abc5392cc158e9b820f77fb37e0f3d219ca29efd448987d"}
Sep 16 18:43:01.961615 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:01.961555    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s" event={"ID":"65f47594-f4f8-4a74-908b-1f7810167c04","Type":"ContainerStarted","Data":"df6e35963b7fa05d37cf440624a4aead4ac6b47950bbc9edcc492473d384310f"}
Sep 16 18:43:01.964965 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:01.964711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76" event={"ID":"6638b647-14f2-4098-8ce9-06175618d722","Type":"ContainerStarted","Data":"d67d3231f3985e9662600884bb4e6612c2017c9d8eadfd54aeb48a3f05597241"}
Sep 16 18:43:02.645875 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.645323    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:43:02.649336 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.649223    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.651922 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.651888    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:43:02.654925 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.654865    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:43:02.654925 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.654877    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:43:02.655222 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.655202    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:43:02.655575 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.655352    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:43:02.655575 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.655447    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:43:02.655575 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.655211    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:43:02.655575 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.655559    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:43:02.655850 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.655580    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:43:02.656012 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.655995    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:43:02.656119 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.655996    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r8mcf\""
Sep 16 18:43:02.656194 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.656176    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-84prt5n5a5cg0\""
Sep 16 18:43:02.656299 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.656049    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:43:02.656759 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.656734    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:43:02.657985 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.657952    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:43:02.661252 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.661232    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:43:02.668540 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.668520    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:43:02.719403 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.719342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.719537 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.719428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29wdv\" (UniqueName: \"kubernetes.io/projected/c9517d5e-8b67-4e44-a514-f7de59564b61-kube-api-access-29wdv\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.719537 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.719462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.719537 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.719492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.719699 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.719590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-web-config\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.719699 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.719659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.719773 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.719724    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.719773 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.719744    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.719773 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.719759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.719937 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.719787    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.719937 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.719838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.720019 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.719942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.720051 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.720015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9517d5e-8b67-4e44-a514-f7de59564b61-config-out\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.720051 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.720045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.720122 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.720093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-config\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.720157 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.720117    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.720157 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.720147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.720226 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.720178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.720226 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.720203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.720324 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.720263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9517d5e-8b67-4e44-a514-f7de59564b61-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822075 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822245 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29wdv\" (UniqueName: \"kubernetes.io/projected/c9517d5e-8b67-4e44-a514-f7de59564b61-kube-api-access-29wdv\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822245 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822245 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822245 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-web-config\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822462 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822462 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822462 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822462 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822462 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822462 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822462 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822882 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822467    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9517d5e-8b67-4e44-a514-f7de59564b61-config-out\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822882 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822882 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822528    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-config\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822882 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822548    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822882 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822882 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822632    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822882 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.822882 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.822686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9517d5e-8b67-4e44-a514-f7de59564b61-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.827713 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.827679    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.828386 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.828007    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.828386 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.828039    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.830645 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.829413    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.830645 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.830165    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.830645 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.830595    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.830893 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.830869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.831622 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.831316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.831622 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.831357    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.831849 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.831824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.832535 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.832495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-config\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.833583 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.833084    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9517d5e-8b67-4e44-a514-f7de59564b61-config-out\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.833583 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.833539    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-web-config\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.834511 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.834472    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.835575 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.834931    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.835575 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.835186    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9517d5e-8b67-4e44-a514-f7de59564b61-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.835575 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.835566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.836330 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.836309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29wdv\" (UniqueName: \"kubernetes.io/projected/c9517d5e-8b67-4e44-a514-f7de59564b61-kube-api-access-29wdv\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.836435 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.836350    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.836751 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.836721    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:02.963965 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:02.963929    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:03.909493 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.909399    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:43:03.912135 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:43:03.912111    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9517d5e_8b67_4e44_a514_f7de59564b61.slice/crio-37101b134198b2754cac186f51be6c93aaa6c39b44c45acaf8388f76c5893d7e WatchSource:0}: Error finding container 37101b134198b2754cac186f51be6c93aaa6c39b44c45acaf8388f76c5893d7e: Status 404 returned error can't find the container with id 37101b134198b2754cac186f51be6c93aaa6c39b44c45acaf8388f76c5893d7e
Sep 16 18:43:03.981413 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.980106    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh" event={"ID":"ff866591-f0fb-4b72-b107-92009c38c858","Type":"ContainerStarted","Data":"1986e416f46bf4bcea67c15a8095a81230c5ba9222f45f5af5a399473c0f38a4"}
Sep 16 18:43:03.981413 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.980744    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"
Sep 16 18:43:03.985626 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.985512    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s" event={"ID":"65f47594-f4f8-4a74-908b-1f7810167c04","Type":"ContainerStarted","Data":"f489abf7597f40ac49fdc47b9fc8be2fddf2ebb257235c0901563b46d2254bcc"}
Sep 16 18:43:03.985626 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.985564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s" event={"ID":"65f47594-f4f8-4a74-908b-1f7810167c04","Type":"ContainerStarted","Data":"da6f82324f9c0eb4fd1e30e21eda8d05eea2ea886e9a6ba9c7c59971c0d8d2e3"}
Sep 16 18:43:03.985626 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.985590    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s" event={"ID":"65f47594-f4f8-4a74-908b-1f7810167c04","Type":"ContainerStarted","Data":"cfc9e58fddd7afe2520c159686df7f301349db9c52014850c182fad62f7ca503"}
Sep 16 18:43:03.986169 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.985981    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:43:03.987653 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.987598    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerStarted","Data":"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"}
Sep 16 18:43:03.987753 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.987698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerStarted","Data":"37101b134198b2754cac186f51be6c93aaa6c39b44c45acaf8388f76c5893d7e"}
Sep 16 18:43:03.988514 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.988484    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh"
Sep 16 18:43:03.991204 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.991184    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76" event={"ID":"6638b647-14f2-4098-8ce9-06175618d722","Type":"ContainerStarted","Data":"87feaaabbfcff1e2fd2a9acd8f151e7c7ecd71ca8859d71cd509fda6a8f2e645"}
Sep 16 18:43:03.994151 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.994133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerStarted","Data":"6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1"}
Sep 16 18:43:03.994244 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.994194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerStarted","Data":"d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da"}
Sep 16 18:43:03.994244 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.994237    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerStarted","Data":"b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5"}
Sep 16 18:43:03.994342 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:03.994276    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerStarted","Data":"505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984"}
Sep 16 18:43:04.008620 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:04.008543    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9gpnh" podStartSLOduration=1.889468454 podStartE2EDuration="4.008530663s" totalImagesPullingTime="2.119062209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:04.00702639 +0000 UTC m=+179.194402176" watchObservedRunningTime="2026-09-16 18:43:04.008530663 +0000 UTC m=+179.195906450"
Sep 16 18:43:04.035840 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:04.035799    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s" podStartSLOduration=1.4170162099999999 podStartE2EDuration="5.035782628s" totalImagesPullingTime="3.618766418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:04.034528675 +0000 UTC m=+179.221904462" watchObservedRunningTime="2026-09-16 18:43:04.035782628 +0000 UTC m=+179.223158416"
Sep 16 18:43:04.062679 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:04.062641    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76" podStartSLOduration=1.942743631 podStartE2EDuration="4.062629072s" totalImagesPullingTime="2.119885441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:04.061026001 +0000 UTC m=+179.248401789" watchObservedRunningTime="2026-09-16 18:43:04.062629072 +0000 UTC m=+179.250004859"
Sep 16 18:43:05.000776 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:05.000740    2582 generic.go:356] "Generic (PLEG): container finished" podID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerID="b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028" exitCode=0
Sep 16 18:43:05.001221 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:05.000842    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerDied","Data":"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"}
Sep 16 18:43:05.004799 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:05.004764    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerStarted","Data":"3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363"}
Sep 16 18:43:05.004921 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:05.004858    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerStarted","Data":"e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f"}
Sep 16 18:43:05.069998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:05.069934    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.139135008 podStartE2EDuration="9.069897327s" totalImagesPullingTime="3.898999741s" totalInitContainerRuntime="31.762578ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:05.06838187 +0000 UTC m=+180.255757656" watchObservedRunningTime="2026-09-16 18:43:05.069897327 +0000 UTC m=+180.257273119"
Sep 16 18:43:05.654541 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:05.654505    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:43:05.654715 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:05.654570    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:43:05.659737 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:05.659709    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:43:06.014960 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.014933    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:43:06.167622 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.167586    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m" podUID="d78a830a-17d2-4940-be08-149b0b789bbb" containerName="registry" containerID="cri-o://f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321" gracePeriod=30
Sep 16 18:43:06.469784 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.469757    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:43:06.561268 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.561202    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d78a830a-17d2-4940-be08-149b0b789bbb-installation-pull-secrets\") pod \"d78a830a-17d2-4940-be08-149b0b789bbb\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") "
Sep 16 18:43:06.561268 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.561263    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d78a830a-17d2-4940-be08-149b0b789bbb-ca-trust-extracted\") pod \"d78a830a-17d2-4940-be08-149b0b789bbb\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") "
Sep 16 18:43:06.561463 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.561308    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls\") pod \"d78a830a-17d2-4940-be08-149b0b789bbb\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") "
Sep 16 18:43:06.561463 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.561351    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d78a830a-17d2-4940-be08-149b0b789bbb-registry-certificates\") pod \"d78a830a-17d2-4940-be08-149b0b789bbb\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") "
Sep 16 18:43:06.561463 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.561377    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d78a830a-17d2-4940-be08-149b0b789bbb-trusted-ca\") pod \"d78a830a-17d2-4940-be08-149b0b789bbb\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") "
Sep 16 18:43:06.561463 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.561402    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-bound-sa-token\") pod \"d78a830a-17d2-4940-be08-149b0b789bbb\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") "
Sep 16 18:43:06.561463 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.561439    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mgcv\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-kube-api-access-6mgcv\") pod \"d78a830a-17d2-4940-be08-149b0b789bbb\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") "
Sep 16 18:43:06.561700 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.561471    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d78a830a-17d2-4940-be08-149b0b789bbb-image-registry-private-configuration\") pod \"d78a830a-17d2-4940-be08-149b0b789bbb\" (UID: \"d78a830a-17d2-4940-be08-149b0b789bbb\") "
Sep 16 18:43:06.562229 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.561983    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78a830a-17d2-4940-be08-149b0b789bbb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d78a830a-17d2-4940-be08-149b0b789bbb" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:06.562394 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.562350    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78a830a-17d2-4940-be08-149b0b789bbb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d78a830a-17d2-4940-be08-149b0b789bbb" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:06.564110 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.564086    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d78a830a-17d2-4940-be08-149b0b789bbb" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:06.564235 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.564176    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d78a830a-17d2-4940-be08-149b0b789bbb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d78a830a-17d2-4940-be08-149b0b789bbb" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:06.564538 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.564510    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d78a830a-17d2-4940-be08-149b0b789bbb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d78a830a-17d2-4940-be08-149b0b789bbb" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:06.564621 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.564560    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-kube-api-access-6mgcv" (OuterVolumeSpecName: "kube-api-access-6mgcv") pod "d78a830a-17d2-4940-be08-149b0b789bbb" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb"). InnerVolumeSpecName "kube-api-access-6mgcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:06.564712 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.564697    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d78a830a-17d2-4940-be08-149b0b789bbb" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:06.572014 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.571993    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78a830a-17d2-4940-be08-149b0b789bbb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d78a830a-17d2-4940-be08-149b0b789bbb" (UID: "d78a830a-17d2-4940-be08-149b0b789bbb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:06.662745 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.662721    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d78a830a-17d2-4940-be08-149b0b789bbb-installation-pull-secrets\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.662745 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.662745    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d78a830a-17d2-4940-be08-149b0b789bbb-ca-trust-extracted\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.662863 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.662754    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-registry-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.662863 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.662769    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d78a830a-17d2-4940-be08-149b0b789bbb-registry-certificates\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.662863 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.662782    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d78a830a-17d2-4940-be08-149b0b789bbb-trusted-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.662863 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.662797    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-bound-sa-token\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.662863 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.662810    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mgcv\" (UniqueName: \"kubernetes.io/projected/d78a830a-17d2-4940-be08-149b0b789bbb-kube-api-access-6mgcv\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.662863 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:06.662852    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d78a830a-17d2-4940-be08-149b0b789bbb-image-registry-private-configuration\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:07.016189 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:07.016137    2582 generic.go:356] "Generic (PLEG): container finished" podID="d78a830a-17d2-4940-be08-149b0b789bbb" containerID="f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321" exitCode=0
Sep 16 18:43:07.016602 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:07.016510    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m"
Sep 16 18:43:07.016717 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:07.016686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m" event={"ID":"d78a830a-17d2-4940-be08-149b0b789bbb","Type":"ContainerDied","Data":"f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321"}
Sep 16 18:43:07.016800 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:07.016748    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd47fb98c-2v75m" event={"ID":"d78a830a-17d2-4940-be08-149b0b789bbb","Type":"ContainerDied","Data":"d465c747fb5a52a95fa6a4a5ce189f754dd9d6db3d92590e0057633fa0453d22"}
Sep 16 18:43:07.016800 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:07.016789    2582 scope.go:117] "RemoveContainer" containerID="f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321"
Sep 16 18:43:07.064781 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:07.064752    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dd47fb98c-2v75m"]
Sep 16 18:43:07.069073 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:07.069046    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6dd47fb98c-2v75m"]
Sep 16 18:43:07.298810 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:07.298734    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d78a830a-17d2-4940-be08-149b0b789bbb" path="/var/lib/kubelet/pods/d78a830a-17d2-4940-be08-149b0b789bbb/volumes"
Sep 16 18:43:07.935663 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:07.935506    2582 scope.go:117] "RemoveContainer" containerID="f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321"
Sep 16 18:43:07.935872 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:43:07.935854    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321\": container with ID starting with f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321 not found: ID does not exist" containerID="f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321"
Sep 16 18:43:07.935934 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:07.935883    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321"} err="failed to get container status \"f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321\": rpc error: code = NotFound desc = could not find container \"f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321\": container with ID starting with f32b54e6445b7858fae12515eb6c5b7b847ba52fe3a93cf4e831efd413efb321 not found: ID does not exist"
Sep 16 18:43:09.033879 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:09.033841    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerStarted","Data":"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"}
Sep 16 18:43:09.034333 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:09.033935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerStarted","Data":"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"}
Sep 16 18:43:09.034333 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:09.033969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerStarted","Data":"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"}
Sep 16 18:43:09.034333 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:09.033994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerStarted","Data":"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"}
Sep 16 18:43:09.034333 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:09.034018    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerStarted","Data":"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"}
Sep 16 18:43:09.034333 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:09.034055    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerStarted","Data":"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"}
Sep 16 18:43:09.070159 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:09.070094    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.063121298 podStartE2EDuration="7.070074357s" totalImagesPullingTime="2.982950256s" totalInitContainerRuntime="24.002803ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:09.067798266 +0000 UTC m=+184.255174053" watchObservedRunningTime="2026-09-16 18:43:09.070074357 +0000 UTC m=+184.257450148"
Sep 16 18:43:10.011606 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:10.011579    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c5f686858-5wp7s"
Sep 16 18:43:12.964737 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:12.964693    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:13.834646 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:13.834614    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fdf7f869b-rk5s8"]
Sep 16 18:43:20.886245 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:20.886213    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:20.886726 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:20.886268    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:38.855254 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:38.855204    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fdf7f869b-rk5s8" podUID="36658226-bed8-4eef-b166-2a8b2425346e" containerName="console" containerID="cri-o://98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82" gracePeriod=15
Sep 16 18:43:39.097393 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.097372    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:43:39.115624 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.115580    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36658226-bed8-4eef-b166-2a8b2425346e-console-oauth-config\") pod \"36658226-bed8-4eef-b166-2a8b2425346e\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") "
Sep 16 18:43:39.115624 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.115610    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-console-config\") pod \"36658226-bed8-4eef-b166-2a8b2425346e\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") "
Sep 16 18:43:39.115624 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.115629    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-oauth-serving-cert\") pod \"36658226-bed8-4eef-b166-2a8b2425346e\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") "
Sep 16 18:43:39.115830 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.115672    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36658226-bed8-4eef-b166-2a8b2425346e-console-serving-cert\") pod \"36658226-bed8-4eef-b166-2a8b2425346e\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") "
Sep 16 18:43:39.115830 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.115702    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zxvw\" (UniqueName: \"kubernetes.io/projected/36658226-bed8-4eef-b166-2a8b2425346e-kube-api-access-2zxvw\") pod \"36658226-bed8-4eef-b166-2a8b2425346e\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") "
Sep 16 18:43:39.115830 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.115721    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-service-ca\") pod \"36658226-bed8-4eef-b166-2a8b2425346e\" (UID: \"36658226-bed8-4eef-b166-2a8b2425346e\") "
Sep 16 18:43:39.116117 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.116081    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-console-config" (OuterVolumeSpecName: "console-config") pod "36658226-bed8-4eef-b166-2a8b2425346e" (UID: "36658226-bed8-4eef-b166-2a8b2425346e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:39.116231 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.116120    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "36658226-bed8-4eef-b166-2a8b2425346e" (UID: "36658226-bed8-4eef-b166-2a8b2425346e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:39.116231 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.116198    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-service-ca" (OuterVolumeSpecName: "service-ca") pod "36658226-bed8-4eef-b166-2a8b2425346e" (UID: "36658226-bed8-4eef-b166-2a8b2425346e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:39.117921 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.117884    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36658226-bed8-4eef-b166-2a8b2425346e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "36658226-bed8-4eef-b166-2a8b2425346e" (UID: "36658226-bed8-4eef-b166-2a8b2425346e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:39.118020 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.117950    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36658226-bed8-4eef-b166-2a8b2425346e-kube-api-access-2zxvw" (OuterVolumeSpecName: "kube-api-access-2zxvw") pod "36658226-bed8-4eef-b166-2a8b2425346e" (UID: "36658226-bed8-4eef-b166-2a8b2425346e"). InnerVolumeSpecName "kube-api-access-2zxvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:39.118020 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.117972    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36658226-bed8-4eef-b166-2a8b2425346e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "36658226-bed8-4eef-b166-2a8b2425346e" (UID: "36658226-bed8-4eef-b166-2a8b2425346e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:39.171537 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.171506    2582 generic.go:356] "Generic (PLEG): container finished" podID="36658226-bed8-4eef-b166-2a8b2425346e" containerID="98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82" exitCode=0
Sep 16 18:43:39.171632 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.171548    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fdf7f869b-rk5s8" event={"ID":"36658226-bed8-4eef-b166-2a8b2425346e","Type":"ContainerDied","Data":"98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82"}
Sep 16 18:43:39.171632 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.171569    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fdf7f869b-rk5s8"
Sep 16 18:43:39.171719 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.171640    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fdf7f869b-rk5s8" event={"ID":"36658226-bed8-4eef-b166-2a8b2425346e","Type":"ContainerDied","Data":"0ae5457adb92dd200fa918a2c31194d507c2f36fc10a86088b26464192749ffa"}
Sep 16 18:43:39.171719 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.171666    2582 scope.go:117] "RemoveContainer" containerID="98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82"
Sep 16 18:43:39.180047 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.180031    2582 scope.go:117] "RemoveContainer" containerID="98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82"
Sep 16 18:43:39.180314 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:43:39.180297    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82\": container with ID starting with 98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82 not found: ID does not exist" containerID="98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82"
Sep 16 18:43:39.180375 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.180327    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82"} err="failed to get container status \"98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82\": rpc error: code = NotFound desc = could not find container \"98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82\": container with ID starting with 98458a90216d5ca0b238cb8f5ae1c37d6e35af0d94cf04ae76d79c7699747e82 not found: ID does not exist"
Sep 16 18:43:39.194377 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.194349    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fdf7f869b-rk5s8"]
Sep 16 18:43:39.199890 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.199872    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fdf7f869b-rk5s8"]
Sep 16 18:43:39.216900 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.216885    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36658226-bed8-4eef-b166-2a8b2425346e-console-serving-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:39.216992 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.216923    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zxvw\" (UniqueName: \"kubernetes.io/projected/36658226-bed8-4eef-b166-2a8b2425346e-kube-api-access-2zxvw\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:39.216992 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.216939    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-service-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:39.216992 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.216951    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36658226-bed8-4eef-b166-2a8b2425346e-console-oauth-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:39.216992 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.216960    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-console-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:39.216992 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.216968    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36658226-bed8-4eef-b166-2a8b2425346e-oauth-serving-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:43:39.295847 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:39.295822    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36658226-bed8-4eef-b166-2a8b2425346e" path="/var/lib/kubelet/pods/36658226-bed8-4eef-b166-2a8b2425346e/volumes"
Sep 16 18:43:40.890681 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:40.890651    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:43:40.894548 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:43:40.894528    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-644fdd7df9-k8t76"
Sep 16 18:44:02.964138 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:02.964104    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:02.986177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:02.986143    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:03.296153 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:03.296085    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:16.703795 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:16.703757    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:44:16.704261 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:16.704150    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="alertmanager" containerID="cri-o://505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984" gracePeriod=120
Sep 16 18:44:16.704339 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:16.704202    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="kube-rbac-proxy-metric" containerID="cri-o://e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f" gracePeriod=120
Sep 16 18:44:16.704398 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:16.704312    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="kube-rbac-proxy" containerID="cri-o://6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1" gracePeriod=120
Sep 16 18:44:16.704450 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:16.704256    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="config-reloader" containerID="cri-o://b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5" gracePeriod=120
Sep 16 18:44:16.704450 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:16.704277    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="prom-label-proxy" containerID="cri-o://3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363" gracePeriod=120
Sep 16 18:44:16.704557 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:16.704259    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="kube-rbac-proxy-web" containerID="cri-o://d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da" gracePeriod=120
Sep 16 18:44:17.102269 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.102187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:44:17.114269 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.114238    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47c4bc73-2eff-41ea-91fc-5acce1bcd526-metrics-certs\") pod \"network-metrics-daemon-2bzzg\" (UID: \"47c4bc73-2eff-41ea-91fc-5acce1bcd526\") " pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:44:17.195190 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.195167    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-564xp\""
Sep 16 18:44:17.202963 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.202928    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bzzg"
Sep 16 18:44:17.343046 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.343018    2582 generic.go:356] "Generic (PLEG): container finished" podID="c578544b-7ad9-4218-bd38-0394eeb17165" containerID="3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363" exitCode=0
Sep 16 18:44:17.343046 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.343040    2582 generic.go:356] "Generic (PLEG): container finished" podID="c578544b-7ad9-4218-bd38-0394eeb17165" containerID="6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1" exitCode=0
Sep 16 18:44:17.343046 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.343047    2582 generic.go:356] "Generic (PLEG): container finished" podID="c578544b-7ad9-4218-bd38-0394eeb17165" containerID="b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5" exitCode=0
Sep 16 18:44:17.343046 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.343053    2582 generic.go:356] "Generic (PLEG): container finished" podID="c578544b-7ad9-4218-bd38-0394eeb17165" containerID="505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984" exitCode=0
Sep 16 18:44:17.343307 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.343070    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerDied","Data":"3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363"}
Sep 16 18:44:17.343307 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.343119    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerDied","Data":"6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1"}
Sep 16 18:44:17.343307 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.343144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerDied","Data":"b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5"}
Sep 16 18:44:17.343307 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.343170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerDied","Data":"505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984"}
Sep 16 18:44:17.475882 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.475857    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2bzzg"]
Sep 16 18:44:17.478463 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:44:17.478440    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c4bc73_2eff_41ea_91fc_5acce1bcd526.slice/crio-3e64f4c20c51e8fab127c7065fbb7a4d9eff043f24d07129c16c470d032dc0c0 WatchSource:0}: Error finding container 3e64f4c20c51e8fab127c7065fbb7a4d9eff043f24d07129c16c470d032dc0c0: Status 404 returned error can't find the container with id 3e64f4c20c51e8fab127c7065fbb7a4d9eff043f24d07129c16c470d032dc0c0
Sep 16 18:44:17.966237 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:17.966213    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.110565 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.110492    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c578544b-7ad9-4218-bd38-0394eeb17165-alertmanager-trusted-ca-bundle\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.110565 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.110536    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c578544b-7ad9-4218-bd38-0394eeb17165-alertmanager-main-db\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.110565 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.110563    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-cluster-tls-config\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.110826 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.110700    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c578544b-7ad9-4218-bd38-0394eeb17165-metrics-client-ca\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.110826 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.110753    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c578544b-7ad9-4218-bd38-0394eeb17165-config-out\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.110826 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.110794    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c578544b-7ad9-4218-bd38-0394eeb17165-tls-assets\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.111012 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.110850    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.111012 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.110896    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-config-volume\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.111012 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.110957    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy-web\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.111012 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.110985    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-main-tls\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.111213 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.111012    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-web-config\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.111213 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.111042    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg6x6\" (UniqueName: \"kubernetes.io/projected/c578544b-7ad9-4218-bd38-0394eeb17165-kube-api-access-zg6x6\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.111213 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.111081    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy\") pod \"c578544b-7ad9-4218-bd38-0394eeb17165\" (UID: \"c578544b-7ad9-4218-bd38-0394eeb17165\") "
Sep 16 18:44:18.113084 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.110849    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c578544b-7ad9-4218-bd38-0394eeb17165-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:18.113258 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.110950    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c578544b-7ad9-4218-bd38-0394eeb17165-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:18.113258 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.111170    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c578544b-7ad9-4218-bd38-0394eeb17165-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:18.114596 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.114553    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:18.114792 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.114753    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:18.115494 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.115462    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c578544b-7ad9-4218-bd38-0394eeb17165-config-out" (OuterVolumeSpecName: "config-out") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:18.115673 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.115641    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c578544b-7ad9-4218-bd38-0394eeb17165-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:18.115899 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.115873    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:18.116268 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.116230    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:18.116448 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.116423    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c578544b-7ad9-4218-bd38-0394eeb17165-kube-api-access-zg6x6" (OuterVolumeSpecName: "kube-api-access-zg6x6") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "kube-api-access-zg6x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:18.117192 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.117167    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-config-volume" (OuterVolumeSpecName: "config-volume") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:18.119788 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.119752    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:18.127834 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.127799    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-web-config" (OuterVolumeSpecName: "web-config") pod "c578544b-7ad9-4218-bd38-0394eeb17165" (UID: "c578544b-7ad9-4218-bd38-0394eeb17165"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:18.212011 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.211987    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.212011 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.212012    2582 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-config-volume\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.212177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.212026    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.212177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.212039    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-main-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.212177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.212053    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-web-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.212177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.212065    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg6x6\" (UniqueName: \"kubernetes.io/projected/c578544b-7ad9-4218-bd38-0394eeb17165-kube-api-access-zg6x6\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.212177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.212076    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.212177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.212089    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c578544b-7ad9-4218-bd38-0394eeb17165-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.212177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.212104    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c578544b-7ad9-4218-bd38-0394eeb17165-alertmanager-main-db\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.212177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.212116    2582 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c578544b-7ad9-4218-bd38-0394eeb17165-cluster-tls-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.212177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.212128    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c578544b-7ad9-4218-bd38-0394eeb17165-metrics-client-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.212177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.212141    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c578544b-7ad9-4218-bd38-0394eeb17165-config-out\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.212177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.212154    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c578544b-7ad9-4218-bd38-0394eeb17165-tls-assets\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.348938 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.348889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2bzzg" event={"ID":"47c4bc73-2eff-41ea-91fc-5acce1bcd526","Type":"ContainerStarted","Data":"3e64f4c20c51e8fab127c7065fbb7a4d9eff043f24d07129c16c470d032dc0c0"}
Sep 16 18:44:18.352522 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.352498    2582 generic.go:356] "Generic (PLEG): container finished" podID="c578544b-7ad9-4218-bd38-0394eeb17165" containerID="e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f" exitCode=0
Sep 16 18:44:18.352522 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.352518    2582 generic.go:356] "Generic (PLEG): container finished" podID="c578544b-7ad9-4218-bd38-0394eeb17165" containerID="d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da" exitCode=0
Sep 16 18:44:18.352677 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.352537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerDied","Data":"e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f"}
Sep 16 18:44:18.352677 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.352600    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerDied","Data":"d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da"}
Sep 16 18:44:18.352677 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.352602    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.352677 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.352635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c578544b-7ad9-4218-bd38-0394eeb17165","Type":"ContainerDied","Data":"df8ae7a620882ce4689d75a65b68e98f1918a28fed39f35916c1d9b355c00abe"}
Sep 16 18:44:18.352878 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.352674    2582 scope.go:117] "RemoveContainer" containerID="3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363"
Sep 16 18:44:18.361320 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.361300    2582 scope.go:117] "RemoveContainer" containerID="e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f"
Sep 16 18:44:18.369440 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.369420    2582 scope.go:117] "RemoveContainer" containerID="6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1"
Sep 16 18:44:18.377743 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.377722    2582 scope.go:117] "RemoveContainer" containerID="d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da"
Sep 16 18:44:18.384276 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.384248    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:44:18.389745 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.389727    2582 scope.go:117] "RemoveContainer" containerID="b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5"
Sep 16 18:44:18.394474 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.394453    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:44:18.400016 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.400000    2582 scope.go:117] "RemoveContainer" containerID="505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984"
Sep 16 18:44:18.409234 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.409219    2582 scope.go:117] "RemoveContainer" containerID="47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557"
Sep 16 18:44:18.423876 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.423860    2582 scope.go:117] "RemoveContainer" containerID="3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363"
Sep 16 18:44:18.424149 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:18.424128    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363\": container with ID starting with 3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363 not found: ID does not exist" containerID="3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363"
Sep 16 18:44:18.424226 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.424155    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363"} err="failed to get container status \"3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363\": rpc error: code = NotFound desc = could not find container \"3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363\": container with ID starting with 3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363 not found: ID does not exist"
Sep 16 18:44:18.424226 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.424172    2582 scope.go:117] "RemoveContainer" containerID="e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f"
Sep 16 18:44:18.424407 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:18.424390    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f\": container with ID starting with e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f not found: ID does not exist" containerID="e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f"
Sep 16 18:44:18.424471 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.424412    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f"} err="failed to get container status \"e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f\": rpc error: code = NotFound desc = could not find container \"e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f\": container with ID starting with e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f not found: ID does not exist"
Sep 16 18:44:18.424471 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.424425    2582 scope.go:117] "RemoveContainer" containerID="6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1"
Sep 16 18:44:18.428155 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:18.426956    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1\": container with ID starting with 6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1 not found: ID does not exist" containerID="6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1"
Sep 16 18:44:18.428155 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.427002    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1"} err="failed to get container status \"6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1\": rpc error: code = NotFound desc = could not find container \"6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1\": container with ID starting with 6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1 not found: ID does not exist"
Sep 16 18:44:18.428155 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.427026    2582 scope.go:117] "RemoveContainer" containerID="d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da"
Sep 16 18:44:18.432121 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:18.432072    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da\": container with ID starting with d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da not found: ID does not exist" containerID="d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da"
Sep 16 18:44:18.432213 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.432129    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da"} err="failed to get container status \"d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da\": rpc error: code = NotFound desc = could not find container \"d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da\": container with ID starting with d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da not found: ID does not exist"
Sep 16 18:44:18.432213 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.432159    2582 scope.go:117] "RemoveContainer" containerID="b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5"
Sep 16 18:44:18.432857 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:18.432834    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5\": container with ID starting with b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5 not found: ID does not exist" containerID="b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5"
Sep 16 18:44:18.432958 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.432865    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5"} err="failed to get container status \"b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5\": rpc error: code = NotFound desc = could not find container \"b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5\": container with ID starting with b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5 not found: ID does not exist"
Sep 16 18:44:18.432958 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.432885    2582 scope.go:117] "RemoveContainer" containerID="505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984"
Sep 16 18:44:18.433310 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:18.433204    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984\": container with ID starting with 505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984 not found: ID does not exist" containerID="505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984"
Sep 16 18:44:18.433310 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.433232    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984"} err="failed to get container status \"505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984\": rpc error: code = NotFound desc = could not find container \"505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984\": container with ID starting with 505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984 not found: ID does not exist"
Sep 16 18:44:18.433310 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.433249    2582 scope.go:117] "RemoveContainer" containerID="47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557"
Sep 16 18:44:18.433811 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:18.433785    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557\": container with ID starting with 47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557 not found: ID does not exist" containerID="47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557"
Sep 16 18:44:18.433872 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.433819    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557"} err="failed to get container status \"47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557\": rpc error: code = NotFound desc = could not find container \"47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557\": container with ID starting with 47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557 not found: ID does not exist"
Sep 16 18:44:18.433872 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.433850    2582 scope.go:117] "RemoveContainer" containerID="3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363"
Sep 16 18:44:18.434087 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434066    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:44:18.434201 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434177    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363"} err="failed to get container status \"3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363\": rpc error: code = NotFound desc = could not find container \"3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363\": container with ID starting with 3ecab30c649ff223a80682d60bb22ec0c27cb9d3882131fc598f481d3981d363 not found: ID does not exist"
Sep 16 18:44:18.434269 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434206    2582 scope.go:117] "RemoveContainer" containerID="e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f"
Sep 16 18:44:18.434491 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434476    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="kube-rbac-proxy-web"
Sep 16 18:44:18.434556 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434497    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="kube-rbac-proxy-web"
Sep 16 18:44:18.434556 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434515    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78a830a-17d2-4940-be08-149b0b789bbb" containerName="registry"
Sep 16 18:44:18.434556 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434524    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78a830a-17d2-4940-be08-149b0b789bbb" containerName="registry"
Sep 16 18:44:18.434556 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434535    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="kube-rbac-proxy-metric"
Sep 16 18:44:18.434556 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434545    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="kube-rbac-proxy-metric"
Sep 16 18:44:18.434748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434565    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="init-config-reloader"
Sep 16 18:44:18.434748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434573    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="init-config-reloader"
Sep 16 18:44:18.434748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434586    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36658226-bed8-4eef-b166-2a8b2425346e" containerName="console"
Sep 16 18:44:18.434748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434594    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="36658226-bed8-4eef-b166-2a8b2425346e" containerName="console"
Sep 16 18:44:18.434748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434605    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="alertmanager"
Sep 16 18:44:18.434748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434523    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f"} err="failed to get container status \"e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f\": rpc error: code = NotFound desc = could not find container \"e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f\": container with ID starting with e7ea64fb598917e557a5d47c1eba008accf28ba71a30b286fd01e5041c129f1f not found: ID does not exist"
Sep 16 18:44:18.434748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434631    2582 scope.go:117] "RemoveContainer" containerID="6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1"
Sep 16 18:44:18.434748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434613    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="alertmanager"
Sep 16 18:44:18.434748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434703    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="kube-rbac-proxy"
Sep 16 18:44:18.434748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434714    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="kube-rbac-proxy"
Sep 16 18:44:18.434748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434735    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="config-reloader"
Sep 16 18:44:18.434748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434743    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="config-reloader"
Sep 16 18:44:18.435207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434754    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="prom-label-proxy"
Sep 16 18:44:18.435207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434774    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="prom-label-proxy"
Sep 16 18:44:18.435207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434864    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="alertmanager"
Sep 16 18:44:18.435207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434894    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78a830a-17d2-4940-be08-149b0b789bbb" containerName="registry"
Sep 16 18:44:18.435207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434933    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="36658226-bed8-4eef-b166-2a8b2425346e" containerName="console"
Sep 16 18:44:18.435207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434942    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="kube-rbac-proxy-web"
Sep 16 18:44:18.435207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434949    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="kube-rbac-proxy-metric"
Sep 16 18:44:18.435207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434984    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="config-reloader"
Sep 16 18:44:18.435207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.434993    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="prom-label-proxy"
Sep 16 18:44:18.435207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.435000    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" containerName="kube-rbac-proxy"
Sep 16 18:44:18.435207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.435008    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1"} err="failed to get container status \"6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1\": rpc error: code = NotFound desc = could not find container \"6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1\": container with ID starting with 6cbf6ca139b24d2701548066f1beb98b66644c0f5b4dd584662529018ed006c1 not found: ID does not exist"
Sep 16 18:44:18.435207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.435025    2582 scope.go:117] "RemoveContainer" containerID="d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da"
Sep 16 18:44:18.435603 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.435286    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da"} err="failed to get container status \"d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da\": rpc error: code = NotFound desc = could not find container \"d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da\": container with ID starting with d85522e4e4167580152c267b1f2756bcfed37b125d437958695c50c5f69ae6da not found: ID does not exist"
Sep 16 18:44:18.435603 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.435301    2582 scope.go:117] "RemoveContainer" containerID="b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5"
Sep 16 18:44:18.435603 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.435557    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5"} err="failed to get container status \"b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5\": rpc error: code = NotFound desc = could not find container \"b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5\": container with ID starting with b8c22dc567c9a484c4b2d3c1480e362d939f3512d69ece9655e45975850421a5 not found: ID does not exist"
Sep 16 18:44:18.435603 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.435571    2582 scope.go:117] "RemoveContainer" containerID="505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984"
Sep 16 18:44:18.435826 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.435808    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984"} err="failed to get container status \"505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984\": rpc error: code = NotFound desc = could not find container \"505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984\": container with ID starting with 505ace5e5102c24d6e9b6a633ac065b0dd0cac7f29a046c373fce4e306983984 not found: ID does not exist"
Sep 16 18:44:18.435894 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.435828    2582 scope.go:117] "RemoveContainer" containerID="47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557"
Sep 16 18:44:18.436121 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.436103    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557"} err="failed to get container status \"47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557\": rpc error: code = NotFound desc = could not find container \"47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557\": container with ID starting with 47d8349d603f80bfd952167ea193520e10df6562408c878f78716efdb43d3557 not found: ID does not exist"
Sep 16 18:44:18.438591 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.438577    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.442527 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.441823    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:44:18.442527 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.442171    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:44:18.442527 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.442416    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:44:18.442731 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.442667    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:44:18.443722 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.442894    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:44:18.443722 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.442945    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:44:18.443722 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.442983    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dcnxc\""
Sep 16 18:44:18.443722 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.443224    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:44:18.443722 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.443472    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:44:18.448178 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.448156    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:44:18.455010 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.454987    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:44:18.515177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515155    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.515296 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.515296 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.515296 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9kx\" (UniqueName: \"kubernetes.io/projected/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-kube-api-access-fg9kx\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.515296 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515269    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.515463 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.515463 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-config-volume\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.515463 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515368    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-config-out\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.515463 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515384    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.515463 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-web-config\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.515608 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.515608 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515484    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.515608 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.515534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616361 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616361 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616476 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616476 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9kx\" (UniqueName: \"kubernetes.io/projected/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-kube-api-access-fg9kx\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616575 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616575 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616677 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-config-volume\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616677 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-config-out\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616677 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616822 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-web-config\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616822 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616822 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.616822 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.616801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.617198 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.617164    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.617324 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.617174    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.618539 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.618500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.619808 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.619749    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-config-volume\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.619953 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.619850    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.620124 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.619893    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.620175 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.620124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.620175 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.620053    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.620175 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.620098    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-config-out\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.620281 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.620187    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.621279 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.621260    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-web-config\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.621501 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.621483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.627182 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.627161    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9kx\" (UniqueName: \"kubernetes.io/projected/fd14d037-f63a-41a2-adaf-1acf0ccd1be4-kube-api-access-fg9kx\") pod \"alertmanager-main-0\" (UID: \"fd14d037-f63a-41a2-adaf-1acf0ccd1be4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:18.750103 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:18.750079    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:19.034557 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:19.034515    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:44:19.037176 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:44:19.037149    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd14d037_f63a_41a2_adaf_1acf0ccd1be4.slice/crio-27fbe571119ef55adcca1419766a8a50987a779b0d4c2dc69b2774826b543499 WatchSource:0}: Error finding container 27fbe571119ef55adcca1419766a8a50987a779b0d4c2dc69b2774826b543499: Status 404 returned error can't find the container with id 27fbe571119ef55adcca1419766a8a50987a779b0d4c2dc69b2774826b543499
Sep 16 18:44:19.297118 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:19.297063    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c578544b-7ad9-4218-bd38-0394eeb17165" path="/var/lib/kubelet/pods/c578544b-7ad9-4218-bd38-0394eeb17165/volumes"
Sep 16 18:44:19.358457 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:19.358428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2bzzg" event={"ID":"47c4bc73-2eff-41ea-91fc-5acce1bcd526","Type":"ContainerStarted","Data":"e3f30776bfbe7548b0f68adaf16a9b86a0b36b64eba01719adc21b44a8f5d65d"}
Sep 16 18:44:19.358575 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:19.358471    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2bzzg" event={"ID":"47c4bc73-2eff-41ea-91fc-5acce1bcd526","Type":"ContainerStarted","Data":"594194e0aaf2e3a93f2f51748a62c302a962fc275ae0e0a3e8706f785c75cfb5"}
Sep 16 18:44:19.360857 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:19.360834    2582 generic.go:356] "Generic (PLEG): container finished" podID="fd14d037-f63a-41a2-adaf-1acf0ccd1be4" containerID="867546a064e95c3d20c53038cecc50d47d5dd7435aa80ea317133e8f5eab7579" exitCode=0
Sep 16 18:44:19.360973 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:19.360876    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd14d037-f63a-41a2-adaf-1acf0ccd1be4","Type":"ContainerDied","Data":"867546a064e95c3d20c53038cecc50d47d5dd7435aa80ea317133e8f5eab7579"}
Sep 16 18:44:19.360973 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:19.360941    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd14d037-f63a-41a2-adaf-1acf0ccd1be4","Type":"ContainerStarted","Data":"27fbe571119ef55adcca1419766a8a50987a779b0d4c2dc69b2774826b543499"}
Sep 16 18:44:19.377595 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:19.377546    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2bzzg" podStartSLOduration=252.583311148 podStartE2EDuration="4m13.377527258s" totalImagesPullingTime="794.21611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:19.375763507 +0000 UTC m=+254.563139294" watchObservedRunningTime="2026-09-16 18:44:19.377527258 +0000 UTC m=+254.564903046"
Sep 16 18:44:20.370548 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.370507    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd14d037-f63a-41a2-adaf-1acf0ccd1be4","Type":"ContainerStarted","Data":"a1243d04aa289b6f5a6968388a25cd7b7506d5b2bacf31e9234da61054aa60fe"}
Sep 16 18:44:20.370883 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.370589    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd14d037-f63a-41a2-adaf-1acf0ccd1be4","Type":"ContainerStarted","Data":"af0ee39cd022cd5ef48fd6cb26f2c3cb60a33086ab3ef02c31b39bb8e2c8b3c5"}
Sep 16 18:44:20.370883 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.370616    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd14d037-f63a-41a2-adaf-1acf0ccd1be4","Type":"ContainerStarted","Data":"4c34904e9752e752a87414644a276a7fb243e8d39403096e702df0f49db5efbb"}
Sep 16 18:44:20.370883 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.370644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd14d037-f63a-41a2-adaf-1acf0ccd1be4","Type":"ContainerStarted","Data":"a980935bb9c98ad78fe183533bd2d12a0c942a00471169d21a1275b423cba881"}
Sep 16 18:44:20.378187 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.378150    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd14d037-f63a-41a2-adaf-1acf0ccd1be4","Type":"ContainerStarted","Data":"87447f73e68246c466354348afeb779439d90eea2cff18bfb77957284190db49"}
Sep 16 18:44:20.378747 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.378699    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd14d037-f63a-41a2-adaf-1acf0ccd1be4","Type":"ContainerStarted","Data":"ed6d0e22ca769895870bd9917ca4e184200a4977a3634fe6392460ddd7601b58"}
Sep 16 18:44:20.408803 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.408754    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.3990025839999998 podStartE2EDuration="2.408738604s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.73602ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:20.406315164 +0000 UTC m=+255.593691014" watchObservedRunningTime="2026-09-16 18:44:20.408738604 +0000 UTC m=+255.596114391"
Sep 16 18:44:20.724521 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.724480    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c794674b7-jnbrv"]
Sep 16 18:44:20.727025 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.726998    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.731442 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.731323    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:44:20.731442 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.731351    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:44:20.731442 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.731386    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:44:20.731677 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.731392    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:44:20.731748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.731730    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-v2bcm\""
Sep 16 18:44:20.731805 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.731746    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:44:20.737381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.737360    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:44:20.745277 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.745256    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c794674b7-jnbrv"]
Sep 16 18:44:20.832360 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.832331    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0ba91952-12c7-4a3c-81a3-00d57a122618-secret-telemeter-client\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.832580 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.832565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0ba91952-12c7-4a3c-81a3-00d57a122618-federate-client-tls\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.832710 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.832694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba91952-12c7-4a3c-81a3-00d57a122618-serving-certs-ca-bundle\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.832834 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.832820    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ba91952-12c7-4a3c-81a3-00d57a122618-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.833001 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.832986    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba91952-12c7-4a3c-81a3-00d57a122618-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.833136 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.833124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0ba91952-12c7-4a3c-81a3-00d57a122618-telemeter-client-tls\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.833256 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.833244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ba91952-12c7-4a3c-81a3-00d57a122618-metrics-client-ca\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.833374 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.833361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ktk\" (UniqueName: \"kubernetes.io/projected/0ba91952-12c7-4a3c-81a3-00d57a122618-kube-api-access-t4ktk\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.934689 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.934649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0ba91952-12c7-4a3c-81a3-00d57a122618-secret-telemeter-client\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.934827 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.934698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0ba91952-12c7-4a3c-81a3-00d57a122618-federate-client-tls\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.934827 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.934726    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba91952-12c7-4a3c-81a3-00d57a122618-serving-certs-ca-bundle\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.934827 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.934755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ba91952-12c7-4a3c-81a3-00d57a122618-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.934827 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.934792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba91952-12c7-4a3c-81a3-00d57a122618-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.934827 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.934824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0ba91952-12c7-4a3c-81a3-00d57a122618-telemeter-client-tls\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.935139 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.934847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ba91952-12c7-4a3c-81a3-00d57a122618-metrics-client-ca\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.935139 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.934871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ktk\" (UniqueName: \"kubernetes.io/projected/0ba91952-12c7-4a3c-81a3-00d57a122618-kube-api-access-t4ktk\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.936604 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.936578    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba91952-12c7-4a3c-81a3-00d57a122618-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.936876 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.936856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba91952-12c7-4a3c-81a3-00d57a122618-serving-certs-ca-bundle\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.937243 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.937207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ba91952-12c7-4a3c-81a3-00d57a122618-metrics-client-ca\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.938092 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.938055    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ba91952-12c7-4a3c-81a3-00d57a122618-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.938617 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.938588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0ba91952-12c7-4a3c-81a3-00d57a122618-telemeter-client-tls\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.938694 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.938665    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0ba91952-12c7-4a3c-81a3-00d57a122618-federate-client-tls\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.938828 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.938811    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0ba91952-12c7-4a3c-81a3-00d57a122618-secret-telemeter-client\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:20.945862 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:20.945833    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ktk\" (UniqueName: \"kubernetes.io/projected/0ba91952-12c7-4a3c-81a3-00d57a122618-kube-api-access-t4ktk\") pod \"telemeter-client-6c794674b7-jnbrv\" (UID: \"0ba91952-12c7-4a3c-81a3-00d57a122618\") " pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:21.039020 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.038962    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv"
Sep 16 18:44:21.059795 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.059771    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:21.060503 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.060110    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="prometheus" containerID="cri-o://ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc" gracePeriod=600
Sep 16 18:44:21.060503 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.060140    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="kube-rbac-proxy" containerID="cri-o://6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00" gracePeriod=600
Sep 16 18:44:21.060503 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.060171    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03" gracePeriod=600
Sep 16 18:44:21.060503 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.060179    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="kube-rbac-proxy-web" containerID="cri-o://afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363" gracePeriod=600
Sep 16 18:44:21.060503 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.060152    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="config-reloader" containerID="cri-o://b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8" gracePeriod=600
Sep 16 18:44:21.060503 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.060473    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="thanos-sidecar" containerID="cri-o://e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2" gracePeriod=600
Sep 16 18:44:21.303122 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.303096    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.338824 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.337422    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c794674b7-jnbrv"]
Sep 16 18:44:21.339927 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:44:21.339878    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba91952_12c7_4a3c_81a3_00d57a122618.slice/crio-a22ed2c1321a97b7d09c7fbe8f9370f58292826097d4faa09ec2e2d6f4c5d947 WatchSource:0}: Error finding container a22ed2c1321a97b7d09c7fbe8f9370f58292826097d4faa09ec2e2d6f4c5d947: Status 404 returned error can't find the container with id a22ed2c1321a97b7d09c7fbe8f9370f58292826097d4faa09ec2e2d6f4c5d947
Sep 16 18:44:21.380097 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.380067    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv" event={"ID":"0ba91952-12c7-4a3c-81a3-00d57a122618","Type":"ContainerStarted","Data":"a22ed2c1321a97b7d09c7fbe8f9370f58292826097d4faa09ec2e2d6f4c5d947"}
Sep 16 18:44:21.383729 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383694    2582 generic.go:356] "Generic (PLEG): container finished" podID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerID="7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03" exitCode=0
Sep 16 18:44:21.383729 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383715    2582 generic.go:356] "Generic (PLEG): container finished" podID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerID="6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00" exitCode=0
Sep 16 18:44:21.383729 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383724    2582 generic.go:356] "Generic (PLEG): container finished" podID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerID="afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363" exitCode=0
Sep 16 18:44:21.383940 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383734    2582 generic.go:356] "Generic (PLEG): container finished" podID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerID="e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2" exitCode=0
Sep 16 18:44:21.383940 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383742    2582 generic.go:356] "Generic (PLEG): container finished" podID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerID="b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8" exitCode=0
Sep 16 18:44:21.383940 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383748    2582 generic.go:356] "Generic (PLEG): container finished" podID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerID="ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc" exitCode=0
Sep 16 18:44:21.383940 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383780    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.383940 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383777    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerDied","Data":"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"}
Sep 16 18:44:21.383940 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383855    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerDied","Data":"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"}
Sep 16 18:44:21.383940 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerDied","Data":"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"}
Sep 16 18:44:21.383940 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383920    2582 scope.go:117] "RemoveContainer" containerID="7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"
Sep 16 18:44:21.384332 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerDied","Data":"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"}
Sep 16 18:44:21.384332 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.383997    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerDied","Data":"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"}
Sep 16 18:44:21.384332 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.384039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerDied","Data":"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"}
Sep 16 18:44:21.384332 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.384081    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c9517d5e-8b67-4e44-a514-f7de59564b61","Type":"ContainerDied","Data":"37101b134198b2754cac186f51be6c93aaa6c39b44c45acaf8388f76c5893d7e"}
Sep 16 18:44:21.391877 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.391855    2582 scope.go:117] "RemoveContainer" containerID="6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"
Sep 16 18:44:21.400078 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.400058    2582 scope.go:117] "RemoveContainer" containerID="afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"
Sep 16 18:44:21.410195 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.410170    2582 scope.go:117] "RemoveContainer" containerID="e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"
Sep 16 18:44:21.418718 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.418690    2582 scope.go:117] "RemoveContainer" containerID="b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"
Sep 16 18:44:21.425548 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.425532    2582 scope.go:117] "RemoveContainer" containerID="ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"
Sep 16 18:44:21.432859 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.432839    2582 scope.go:117] "RemoveContainer" containerID="b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"
Sep 16 18:44:21.439604 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.439583    2582 scope.go:117] "RemoveContainer" containerID="7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"
Sep 16 18:44:21.439752 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.439733    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-trusted-ca-bundle\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.439841 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.439781    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-serving-certs-ca-bundle\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.439841 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.439814    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-2\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440016 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.439839    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-thanos-prometheus-http-client-file\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440016 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.439866    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-metrics-client-certs\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440016 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.439899    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-0\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440016 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.439949    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-1\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440016 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:21.439971    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": container with ID starting with 7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03 not found: ID does not exist" containerID="7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"
Sep 16 18:44:21.440016 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440003    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-web-config\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440301 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440006    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"} err="failed to get container status \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": rpc error: code = NotFound desc = could not find container \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": container with ID starting with 7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03 not found: ID does not exist"
Sep 16 18:44:21.440301 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440032    2582 scope.go:117] "RemoveContainer" containerID="6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"
Sep 16 18:44:21.440301 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440030    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-kubelet-serving-ca-bundle\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440301 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440096    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:21.440301 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440127    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29wdv\" (UniqueName: \"kubernetes.io/projected/c9517d5e-8b67-4e44-a514-f7de59564b61-kube-api-access-29wdv\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440301 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440162    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440301 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440205    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-tls\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440301 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440259    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-grpc-tls\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440301 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440295    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-metrics-client-ca\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440757 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440319    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9517d5e-8b67-4e44-a514-f7de59564b61-config-out\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440757 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440347    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-db\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440757 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440372    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-kube-rbac-proxy\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440757 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440443    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440757 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440468    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-config\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.440757 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440500    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9517d5e-8b67-4e44-a514-f7de59564b61-tls-assets\") pod \"c9517d5e-8b67-4e44-a514-f7de59564b61\" (UID: \"c9517d5e-8b67-4e44-a514-f7de59564b61\") "
Sep 16 18:44:21.441225 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440779    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.441225 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:21.441104    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": container with ID starting with 6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00 not found: ID does not exist" containerID="6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"
Sep 16 18:44:21.441225 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.441136    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"} err="failed to get container status \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": rpc error: code = NotFound desc = could not find container \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": container with ID starting with 6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00 not found: ID does not exist"
Sep 16 18:44:21.441225 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.441159    2582 scope.go:117] "RemoveContainer" containerID="afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"
Sep 16 18:44:21.442330 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:21.441863    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": container with ID starting with afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363 not found: ID does not exist" containerID="afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"
Sep 16 18:44:21.442330 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.441915    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"} err="failed to get container status \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": rpc error: code = NotFound desc = could not find container \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": container with ID starting with afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363 not found: ID does not exist"
Sep 16 18:44:21.442330 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.441941    2582 scope.go:117] "RemoveContainer" containerID="e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"
Sep 16 18:44:21.442330 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440253    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:21.442330 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440322    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:21.442330 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440446    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:21.442330 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.440710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:21.442764 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.442499    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:21.442764 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.442562    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:21.443164 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:21.443137    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": container with ID starting with e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2 not found: ID does not exist" containerID="e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"
Sep 16 18:44:21.443248 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.443171    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"} err="failed to get container status \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": rpc error: code = NotFound desc = could not find container \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": container with ID starting with e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2 not found: ID does not exist"
Sep 16 18:44:21.443248 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.443194    2582 scope.go:117] "RemoveContainer" containerID="b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"
Sep 16 18:44:21.443560 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.443467    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9517d5e-8b67-4e44-a514-f7de59564b61-kube-api-access-29wdv" (OuterVolumeSpecName: "kube-api-access-29wdv") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "kube-api-access-29wdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:21.443651 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:21.443608    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": container with ID starting with b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8 not found: ID does not exist" containerID="b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"
Sep 16 18:44:21.443651 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.443633    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"} err="failed to get container status \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": rpc error: code = NotFound desc = could not find container \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": container with ID starting with b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8 not found: ID does not exist"
Sep 16 18:44:21.444069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.443653    2582 scope.go:117] "RemoveContainer" containerID="ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"
Sep 16 18:44:21.444069 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.443790    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:21.444194 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:21.444057    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": container with ID starting with ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc not found: ID does not exist" containerID="ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"
Sep 16 18:44:21.444194 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.444162    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"} err="failed to get container status \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": rpc error: code = NotFound desc = could not find container \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": container with ID starting with ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc not found: ID does not exist"
Sep 16 18:44:21.444194 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.444181    2582 scope.go:117] "RemoveContainer" containerID="b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"
Sep 16 18:44:21.444629 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.444415    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9517d5e-8b67-4e44-a514-f7de59564b61-config-out" (OuterVolumeSpecName: "config-out") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:21.444629 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.444488    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:21.444629 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:44:21.444524    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": container with ID starting with b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028 not found: ID does not exist" containerID="b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"
Sep 16 18:44:21.444629 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.444549    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9517d5e-8b67-4e44-a514-f7de59564b61-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:21.444629 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.444562    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"} err="failed to get container status \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": rpc error: code = NotFound desc = could not find container \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": container with ID starting with b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028 not found: ID does not exist"
Sep 16 18:44:21.444629 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.444610    2582 scope.go:117] "RemoveContainer" containerID="7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"
Sep 16 18:44:21.445406 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.445073    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:21.445406 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.445351    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:21.445406 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.445355    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"} err="failed to get container status \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": rpc error: code = NotFound desc = could not find container \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": container with ID starting with 7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03 not found: ID does not exist"
Sep 16 18:44:21.445406 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.445381    2582 scope.go:117] "RemoveContainer" containerID="6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"
Sep 16 18:44:21.445864 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.445744    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"} err="failed to get container status \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": rpc error: code = NotFound desc = could not find container \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": container with ID starting with 6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00 not found: ID does not exist"
Sep 16 18:44:21.445864 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.445780    2582 scope.go:117] "RemoveContainer" containerID="afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"
Sep 16 18:44:21.445864 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.445748    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:21.446088 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.445882    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-config" (OuterVolumeSpecName: "config") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:21.446146 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.446085    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"} err="failed to get container status \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": rpc error: code = NotFound desc = could not find container \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": container with ID starting with afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363 not found: ID does not exist"
Sep 16 18:44:21.446146 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.446106    2582 scope.go:117] "RemoveContainer" containerID="e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"
Sep 16 18:44:21.446260 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.446220    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:21.446458 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.446424    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"} err="failed to get container status \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": rpc error: code = NotFound desc = could not find container \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": container with ID starting with e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2 not found: ID does not exist"
Sep 16 18:44:21.446533 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.446460    2582 scope.go:117] "RemoveContainer" containerID="b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"
Sep 16 18:44:21.446533 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.446475    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:21.446630 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.446606    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:21.446737 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.446720    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"} err="failed to get container status \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": rpc error: code = NotFound desc = could not find container \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": container with ID starting with b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8 not found: ID does not exist"
Sep 16 18:44:21.446775 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.446739    2582 scope.go:117] "RemoveContainer" containerID="ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"
Sep 16 18:44:21.447049 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.447020    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"} err="failed to get container status \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": rpc error: code = NotFound desc = could not find container \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": container with ID starting with ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc not found: ID does not exist"
Sep 16 18:44:21.447107 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.447049    2582 scope.go:117] "RemoveContainer" containerID="b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"
Sep 16 18:44:21.447292 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.447275    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"} err="failed to get container status \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": rpc error: code = NotFound desc = could not find container \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": container with ID starting with b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028 not found: ID does not exist"
Sep 16 18:44:21.447357 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.447295    2582 scope.go:117] "RemoveContainer" containerID="7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"
Sep 16 18:44:21.447561 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.447543    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"} err="failed to get container status \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": rpc error: code = NotFound desc = could not find container \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": container with ID starting with 7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03 not found: ID does not exist"
Sep 16 18:44:21.447623 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.447562    2582 scope.go:117] "RemoveContainer" containerID="6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"
Sep 16 18:44:21.447876 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.447851    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"} err="failed to get container status \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": rpc error: code = NotFound desc = could not find container \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": container with ID starting with 6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00 not found: ID does not exist"
Sep 16 18:44:21.447952 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.447882    2582 scope.go:117] "RemoveContainer" containerID="afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"
Sep 16 18:44:21.448219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.448201    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"} err="failed to get container status \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": rpc error: code = NotFound desc = could not find container \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": container with ID starting with afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363 not found: ID does not exist"
Sep 16 18:44:21.448265 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.448220    2582 scope.go:117] "RemoveContainer" containerID="e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"
Sep 16 18:44:21.448456 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.448437    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"} err="failed to get container status \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": rpc error: code = NotFound desc = could not find container \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": container with ID starting with e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2 not found: ID does not exist"
Sep 16 18:44:21.448456 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.448456    2582 scope.go:117] "RemoveContainer" containerID="b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"
Sep 16 18:44:21.448685 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.448662    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"} err="failed to get container status \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": rpc error: code = NotFound desc = could not find container \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": container with ID starting with b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8 not found: ID does not exist"
Sep 16 18:44:21.448735 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.448689    2582 scope.go:117] "RemoveContainer" containerID="ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"
Sep 16 18:44:21.448943 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.448889    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"} err="failed to get container status \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": rpc error: code = NotFound desc = could not find container \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": container with ID starting with ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc not found: ID does not exist"
Sep 16 18:44:21.448943 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.448937    2582 scope.go:117] "RemoveContainer" containerID="b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"
Sep 16 18:44:21.449180 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.449158    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"} err="failed to get container status \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": rpc error: code = NotFound desc = could not find container \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": container with ID starting with b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028 not found: ID does not exist"
Sep 16 18:44:21.449221 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.449183    2582 scope.go:117] "RemoveContainer" containerID="7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"
Sep 16 18:44:21.449455 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.449438    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"} err="failed to get container status \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": rpc error: code = NotFound desc = could not find container \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": container with ID starting with 7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03 not found: ID does not exist"
Sep 16 18:44:21.449507 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.449456    2582 scope.go:117] "RemoveContainer" containerID="6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"
Sep 16 18:44:21.449666 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.449650    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"} err="failed to get container status \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": rpc error: code = NotFound desc = could not find container \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": container with ID starting with 6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00 not found: ID does not exist"
Sep 16 18:44:21.449701 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.449665    2582 scope.go:117] "RemoveContainer" containerID="afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"
Sep 16 18:44:21.449868 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.449850    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"} err="failed to get container status \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": rpc error: code = NotFound desc = could not find container \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": container with ID starting with afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363 not found: ID does not exist"
Sep 16 18:44:21.449935 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.449870    2582 scope.go:117] "RemoveContainer" containerID="e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"
Sep 16 18:44:21.450109 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.450084    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"} err="failed to get container status \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": rpc error: code = NotFound desc = could not find container \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": container with ID starting with e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2 not found: ID does not exist"
Sep 16 18:44:21.450109 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.450108    2582 scope.go:117] "RemoveContainer" containerID="b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"
Sep 16 18:44:21.450347 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.450326    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"} err="failed to get container status \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": rpc error: code = NotFound desc = could not find container \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": container with ID starting with b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8 not found: ID does not exist"
Sep 16 18:44:21.450347 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.450346    2582 scope.go:117] "RemoveContainer" containerID="ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"
Sep 16 18:44:21.450620 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.450602    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"} err="failed to get container status \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": rpc error: code = NotFound desc = could not find container \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": container with ID starting with ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc not found: ID does not exist"
Sep 16 18:44:21.450664 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.450622    2582 scope.go:117] "RemoveContainer" containerID="b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"
Sep 16 18:44:21.450840 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.450825    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"} err="failed to get container status \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": rpc error: code = NotFound desc = could not find container \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": container with ID starting with b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028 not found: ID does not exist"
Sep 16 18:44:21.450880 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.450840    2582 scope.go:117] "RemoveContainer" containerID="7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"
Sep 16 18:44:21.451144 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.451126    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"} err="failed to get container status \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": rpc error: code = NotFound desc = could not find container \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": container with ID starting with 7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03 not found: ID does not exist"
Sep 16 18:44:21.451212 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.451143    2582 scope.go:117] "RemoveContainer" containerID="6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"
Sep 16 18:44:21.451409 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.451393    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"} err="failed to get container status \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": rpc error: code = NotFound desc = could not find container \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": container with ID starting with 6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00 not found: ID does not exist"
Sep 16 18:44:21.451478 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.451410    2582 scope.go:117] "RemoveContainer" containerID="afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"
Sep 16 18:44:21.451668 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.451643    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"} err="failed to get container status \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": rpc error: code = NotFound desc = could not find container \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": container with ID starting with afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363 not found: ID does not exist"
Sep 16 18:44:21.451668 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.451666    2582 scope.go:117] "RemoveContainer" containerID="e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"
Sep 16 18:44:21.451941 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.451919    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"} err="failed to get container status \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": rpc error: code = NotFound desc = could not find container \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": container with ID starting with e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2 not found: ID does not exist"
Sep 16 18:44:21.452020 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.451943    2582 scope.go:117] "RemoveContainer" containerID="b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"
Sep 16 18:44:21.452291 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.452274    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"} err="failed to get container status \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": rpc error: code = NotFound desc = could not find container \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": container with ID starting with b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8 not found: ID does not exist"
Sep 16 18:44:21.452345 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.452293    2582 scope.go:117] "RemoveContainer" containerID="ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"
Sep 16 18:44:21.452522 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.452503    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"} err="failed to get container status \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": rpc error: code = NotFound desc = could not find container \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": container with ID starting with ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc not found: ID does not exist"
Sep 16 18:44:21.452566 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.452523    2582 scope.go:117] "RemoveContainer" containerID="b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"
Sep 16 18:44:21.452742 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.452726    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"} err="failed to get container status \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": rpc error: code = NotFound desc = could not find container \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": container with ID starting with b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028 not found: ID does not exist"
Sep 16 18:44:21.452810 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.452744    2582 scope.go:117] "RemoveContainer" containerID="7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"
Sep 16 18:44:21.453056 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.453027    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03"} err="failed to get container status \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": rpc error: code = NotFound desc = could not find container \"7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03\": container with ID starting with 7872a2ef9e2a9885cf3357889d093b60f485f62300b1745fc9ae7a0784304c03 not found: ID does not exist"
Sep 16 18:44:21.453056 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.453045    2582 scope.go:117] "RemoveContainer" containerID="6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"
Sep 16 18:44:21.453313 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.453296    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00"} err="failed to get container status \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": rpc error: code = NotFound desc = could not find container \"6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00\": container with ID starting with 6180653f5c545b982c68bc09a9fc528179b23f9de038641da937cf0eca727b00 not found: ID does not exist"
Sep 16 18:44:21.453366 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.453314    2582 scope.go:117] "RemoveContainer" containerID="afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"
Sep 16 18:44:21.453564 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.453544    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363"} err="failed to get container status \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": rpc error: code = NotFound desc = could not find container \"afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363\": container with ID starting with afc7693ffd430ecfa65097e7298f4b913fa1b6ae69a346247aba14dcaaf30363 not found: ID does not exist"
Sep 16 18:44:21.453621 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.453564    2582 scope.go:117] "RemoveContainer" containerID="e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"
Sep 16 18:44:21.453811 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.453793    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2"} err="failed to get container status \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": rpc error: code = NotFound desc = could not find container \"e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2\": container with ID starting with e7c43a3d8cf597381ac36d33646f9ed83248e01c91b54f47c1b72f4c58405ef2 not found: ID does not exist"
Sep 16 18:44:21.453811 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.453810    2582 scope.go:117] "RemoveContainer" containerID="b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"
Sep 16 18:44:21.454113 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.454096    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8"} err="failed to get container status \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": rpc error: code = NotFound desc = could not find container \"b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8\": container with ID starting with b30293bd406c0c1835d12086a9dcaacbb0086d7b2e658ef43f97d2a153f381f8 not found: ID does not exist"
Sep 16 18:44:21.454190 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.454120    2582 scope.go:117] "RemoveContainer" containerID="ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"
Sep 16 18:44:21.454367 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.454351    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc"} err="failed to get container status \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": rpc error: code = NotFound desc = could not find container \"ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc\": container with ID starting with ebbe3ccc4554265f145d899e4ca2a56ceec3391adabdab706f2d947fbdff92dc not found: ID does not exist"
Sep 16 18:44:21.454367 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.454367    2582 scope.go:117] "RemoveContainer" containerID="b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"
Sep 16 18:44:21.454633 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.454618    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028"} err="failed to get container status \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": rpc error: code = NotFound desc = could not find container \"b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028\": container with ID starting with b5776231c08a9365dc453c8eee86e04ca3b9c7a1937579f31e098676442e7028 not found: ID does not exist"
Sep 16 18:44:21.457062 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.457041    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-web-config" (OuterVolumeSpecName: "web-config") pod "c9517d5e-8b67-4e44-a514-f7de59564b61" (UID: "c9517d5e-8b67-4e44-a514-f7de59564b61"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541751    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9517d5e-8b67-4e44-a514-f7de59564b61-tls-assets\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541779    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541796    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541810    2582 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541826    2582 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-metrics-client-certs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541840    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541854    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541867    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-web-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541880    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541893    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29wdv\" (UniqueName: \"kubernetes.io/projected/c9517d5e-8b67-4e44-a514-f7de59564b61-kube-api-access-29wdv\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541934    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541961    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541982    2582 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-grpc-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.541998 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.541998    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9517d5e-8b67-4e44-a514-f7de59564b61-configmap-metrics-client-ca\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.542454 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.542011    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9517d5e-8b67-4e44-a514-f7de59564b61-config-out\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.542454 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.542023    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c9517d5e-8b67-4e44-a514-f7de59564b61-prometheus-k8s-db\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.542454 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.542034    2582 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-kube-rbac-proxy\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.542454 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.542048    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.542454 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.542060    2582 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c9517d5e-8b67-4e44-a514-f7de59564b61-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:44:21.713973 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.713948    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:21.725170 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.725149    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:21.756695 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.756675    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:21.756992 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.756978    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="kube-rbac-proxy"
Sep 16 18:44:21.756992 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.756993    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="kube-rbac-proxy"
Sep 16 18:44:21.757089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757004    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="prometheus"
Sep 16 18:44:21.757089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757012    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="prometheus"
Sep 16 18:44:21.757089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757019    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="config-reloader"
Sep 16 18:44:21.757089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757025    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="config-reloader"
Sep 16 18:44:21.757089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757032    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="thanos-sidecar"
Sep 16 18:44:21.757089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757037    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="thanos-sidecar"
Sep 16 18:44:21.757089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757042    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="kube-rbac-proxy-thanos"
Sep 16 18:44:21.757089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757047    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="kube-rbac-proxy-thanos"
Sep 16 18:44:21.757089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757059    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="kube-rbac-proxy-web"
Sep 16 18:44:21.757089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757063    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="kube-rbac-proxy-web"
Sep 16 18:44:21.757089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757072    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="init-config-reloader"
Sep 16 18:44:21.757089 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757076    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="init-config-reloader"
Sep 16 18:44:21.757499 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757120    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="kube-rbac-proxy"
Sep 16 18:44:21.757499 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757128    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="prometheus"
Sep 16 18:44:21.757499 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757135    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="kube-rbac-proxy-thanos"
Sep 16 18:44:21.757499 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757141    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="thanos-sidecar"
Sep 16 18:44:21.757499 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757147    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="config-reloader"
Sep 16 18:44:21.757499 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.757152    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" containerName="kube-rbac-proxy-web"
Sep 16 18:44:21.761992 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.761972    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.765045 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.764819    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:44:21.765045 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.764987    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:44:21.766207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.765324    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:44:21.766207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.765580    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:44:21.766207 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.766189    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:44:21.766438 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.766310    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:44:21.766710 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.766692    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:44:21.768203 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.767453    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r8mcf\""
Sep 16 18:44:21.768422 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.768401    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:44:21.768523 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.768463    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-84prt5n5a5cg0\""
Sep 16 18:44:21.768622 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.768569    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:44:21.768622 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.768588    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:44:21.768773 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.768723    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:44:21.769290 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.769269    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:44:21.770855 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.770834    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:44:21.777604 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.777581    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:44:21.778656 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.778636    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:21.844965 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.844893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.844965 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.844941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845112 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.844967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845112 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.844997    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845112 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845112 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845112 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-web-config\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845112 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845101    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nssg9\" (UniqueName: \"kubernetes.io/projected/41c47de1-f9f0-4188-8685-c7956e7b7e25-kube-api-access-nssg9\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845335 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41c47de1-f9f0-4188-8685-c7956e7b7e25-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845335 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845335 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845335 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845335 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-config\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845335 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845301    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41c47de1-f9f0-4188-8685-c7956e7b7e25-config-out\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845335 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845595 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845344    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845595 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845595 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845595 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.845595 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.845450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.946854 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.946832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-config\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947051 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41c47de1-f9f0-4188-8685-c7956e7b7e25-config-out\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947165 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947265 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947394 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947494 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947494 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947494 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947494 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947641    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-web-config\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nssg9\" (UniqueName: \"kubernetes.io/projected/41c47de1-f9f0-4188-8685-c7956e7b7e25-kube-api-access-nssg9\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.947720 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41c47de1-f9f0-4188-8685-c7956e7b7e25-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.948149 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.948149 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.948149 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.947799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.950232 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.949245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.950232 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.950010    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.951827 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.950720    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.951827 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.951516    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.952888 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.952859    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-config\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.952995 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.952894    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41c47de1-f9f0-4188-8685-c7956e7b7e25-config-out\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.953330 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.953162    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.954188 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.954081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.955205 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.954587    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41c47de1-f9f0-4188-8685-c7956e7b7e25-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.955205 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.954979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-web-config\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.955205 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.955100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.955816 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.955550    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.955816 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.955669    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.955816 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.955726    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.955816 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.955778    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.956097 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.955861    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.956097 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.955941    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.957012 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.956985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41c47de1-f9f0-4188-8685-c7956e7b7e25-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.958111 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.958086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41c47de1-f9f0-4188-8685-c7956e7b7e25-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:21.961367 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:21.961346    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nssg9\" (UniqueName: \"kubernetes.io/projected/41c47de1-f9f0-4188-8685-c7956e7b7e25-kube-api-access-nssg9\") pod \"prometheus-k8s-0\" (UID: \"41c47de1-f9f0-4188-8685-c7956e7b7e25\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:22.078079 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:22.078052    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:22.377394 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:22.377357    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:22.378053 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:44:22.378018    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c47de1_f9f0_4188_8685_c7956e7b7e25.slice/crio-9707db48c263e6f9277bda57eaf0646c5b0a7fc0c76a391ed11abd2260c6354b WatchSource:0}: Error finding container 9707db48c263e6f9277bda57eaf0646c5b0a7fc0c76a391ed11abd2260c6354b: Status 404 returned error can't find the container with id 9707db48c263e6f9277bda57eaf0646c5b0a7fc0c76a391ed11abd2260c6354b
Sep 16 18:44:22.389704 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:22.389679    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41c47de1-f9f0-4188-8685-c7956e7b7e25","Type":"ContainerStarted","Data":"9707db48c263e6f9277bda57eaf0646c5b0a7fc0c76a391ed11abd2260c6354b"}
Sep 16 18:44:23.295779 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:23.295717    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9517d5e-8b67-4e44-a514-f7de59564b61" path="/var/lib/kubelet/pods/c9517d5e-8b67-4e44-a514-f7de59564b61/volumes"
Sep 16 18:44:23.397350 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:23.397309    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv" event={"ID":"0ba91952-12c7-4a3c-81a3-00d57a122618","Type":"ContainerStarted","Data":"6e1009740ac8f54da1e800432a627ce9e13ff59bf68616895790fcdc4870cfd9"}
Sep 16 18:44:23.397701 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:23.397378    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv" event={"ID":"0ba91952-12c7-4a3c-81a3-00d57a122618","Type":"ContainerStarted","Data":"4e8204855accf2ca92000dac8bf27908bf59d61c9ed1f0cd5e43164033bd275b"}
Sep 16 18:44:23.397701 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:23.397407    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv" event={"ID":"0ba91952-12c7-4a3c-81a3-00d57a122618","Type":"ContainerStarted","Data":"bf8ba42e2bc685c9163053f347d519a2cdba8a194901ca85759acfff34f86588"}
Sep 16 18:44:23.398951 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:23.398927    2582 generic.go:356] "Generic (PLEG): container finished" podID="41c47de1-f9f0-4188-8685-c7956e7b7e25" containerID="5d80e431b151accc83f01eb5813cff705466b3ef724619232e698a958a9f174f" exitCode=0
Sep 16 18:44:23.399044 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:23.398978    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41c47de1-f9f0-4188-8685-c7956e7b7e25","Type":"ContainerDied","Data":"5d80e431b151accc83f01eb5813cff705466b3ef724619232e698a958a9f174f"}
Sep 16 18:44:23.423353 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:23.423317    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c794674b7-jnbrv" podStartSLOduration=1.730002373 podStartE2EDuration="3.423302848s" totalImagesPullingTime="1.693300475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:23.422487578 +0000 UTC m=+258.609863382" watchObservedRunningTime="2026-09-16 18:44:23.423302848 +0000 UTC m=+258.610678629"
Sep 16 18:44:24.406968 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:24.406932    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41c47de1-f9f0-4188-8685-c7956e7b7e25","Type":"ContainerStarted","Data":"9674628939cdadcc1cc28f70597ec510a63a3470030b652d96bc2bcbeff1c940"}
Sep 16 18:44:24.407350 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:24.407030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41c47de1-f9f0-4188-8685-c7956e7b7e25","Type":"ContainerStarted","Data":"7156374fd0b95dea1b153ce3bd8db2cbb77632a5deb19119c6287ed7788f24bd"}
Sep 16 18:44:24.407350 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:24.407067    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41c47de1-f9f0-4188-8685-c7956e7b7e25","Type":"ContainerStarted","Data":"b592cd8a79e378798b4efd94b34371e03557670736f5352a8c8b0841bad7134c"}
Sep 16 18:44:24.407350 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:24.407090    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41c47de1-f9f0-4188-8685-c7956e7b7e25","Type":"ContainerStarted","Data":"1b5825ebb727d3973321210a3dae92ecaa4d3f8ee310844af90772f291c85b47"}
Sep 16 18:44:24.407350 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:24.407114    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41c47de1-f9f0-4188-8685-c7956e7b7e25","Type":"ContainerStarted","Data":"aa955ea763563627d1041c1206a280b5b170b55fcae4ae5e1c290e745da08d03"}
Sep 16 18:44:24.407350 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:24.407140    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41c47de1-f9f0-4188-8685-c7956e7b7e25","Type":"ContainerStarted","Data":"07ae15ae907bc3f9509d153d7e107a004d2e39fd680d9afe0533c1c2d8b2df05"}
Sep 16 18:44:24.443170 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:24.443119    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.419880441 podStartE2EDuration="3.443100949s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.220508ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:24.440328842 +0000 UTC m=+259.627704629" watchObservedRunningTime="2026-09-16 18:44:24.443100949 +0000 UTC m=+259.630476995"
Sep 16 18:44:27.078497 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:44:27.078411    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:05.179384 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:45:05.179353    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 18:45:05.181575 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:45:05.181550    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 18:45:05.191820 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:45:05.191795    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:45:22.079100 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:45:22.079065    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:22.097361 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:45:22.097334    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:22.678631 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:45:22.678606    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:50:05.213703 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:50:05.213672    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 18:50:05.214219 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:50:05.213672    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 18:51:20.561232 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.561191    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-zlp5j"]
Sep 16 18:51:20.564893 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.564875    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zlp5j"
Sep 16 18:51:20.567270 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.567252    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:51:20.567393 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.567270    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:51:20.567393 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.567293    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2q5x2\""
Sep 16 18:51:20.567393 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.567330    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:51:20.568112 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.568097    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:51:20.573060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.573040    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zlp5j"]
Sep 16 18:51:20.599516 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.599495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdd9\" (UniqueName: \"kubernetes.io/projected/989138ce-8ff9-4abb-ad44-b07bfb5dcf59-kube-api-access-vzdd9\") pod \"s3-init-zlp5j\" (UID: \"989138ce-8ff9-4abb-ad44-b07bfb5dcf59\") " pod="kserve/s3-init-zlp5j"
Sep 16 18:51:20.599618 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.599548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989138ce-8ff9-4abb-ad44-b07bfb5dcf59-model-cache\") pod \"s3-init-zlp5j\" (UID: \"989138ce-8ff9-4abb-ad44-b07bfb5dcf59\") " pod="kserve/s3-init-zlp5j"
Sep 16 18:51:20.700387 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.700357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdd9\" (UniqueName: \"kubernetes.io/projected/989138ce-8ff9-4abb-ad44-b07bfb5dcf59-kube-api-access-vzdd9\") pod \"s3-init-zlp5j\" (UID: \"989138ce-8ff9-4abb-ad44-b07bfb5dcf59\") " pod="kserve/s3-init-zlp5j"
Sep 16 18:51:20.700528 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.700425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989138ce-8ff9-4abb-ad44-b07bfb5dcf59-model-cache\") pod \"s3-init-zlp5j\" (UID: \"989138ce-8ff9-4abb-ad44-b07bfb5dcf59\") " pod="kserve/s3-init-zlp5j"
Sep 16 18:51:20.700777 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.700757    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989138ce-8ff9-4abb-ad44-b07bfb5dcf59-model-cache\") pod \"s3-init-zlp5j\" (UID: \"989138ce-8ff9-4abb-ad44-b07bfb5dcf59\") " pod="kserve/s3-init-zlp5j"
Sep 16 18:51:20.710194 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.710172    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdd9\" (UniqueName: \"kubernetes.io/projected/989138ce-8ff9-4abb-ad44-b07bfb5dcf59-kube-api-access-vzdd9\") pod \"s3-init-zlp5j\" (UID: \"989138ce-8ff9-4abb-ad44-b07bfb5dcf59\") " pod="kserve/s3-init-zlp5j"
Sep 16 18:51:20.885599 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:20.885547    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zlp5j"
Sep 16 18:51:21.160202 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:21.160133    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zlp5j"]
Sep 16 18:51:21.162677 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:51:21.162650    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod989138ce_8ff9_4abb_ad44_b07bfb5dcf59.slice/crio-b823e06a7bc5d9b9b9f628b037080a6dde4f580750bdb4508d53997edb6d8abc WatchSource:0}: Error finding container b823e06a7bc5d9b9b9f628b037080a6dde4f580750bdb4508d53997edb6d8abc: Status 404 returned error can't find the container with id b823e06a7bc5d9b9b9f628b037080a6dde4f580750bdb4508d53997edb6d8abc
Sep 16 18:51:21.164601 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:21.164583    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:51:21.187309 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:21.187282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zlp5j" event={"ID":"989138ce-8ff9-4abb-ad44-b07bfb5dcf59","Type":"ContainerStarted","Data":"b823e06a7bc5d9b9b9f628b037080a6dde4f580750bdb4508d53997edb6d8abc"}
Sep 16 18:51:25.207407 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:25.207364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zlp5j" event={"ID":"989138ce-8ff9-4abb-ad44-b07bfb5dcf59","Type":"ContainerStarted","Data":"db2538b76f2f08b7aad5461b805b2e8fb6a3cc06b5a6fe7057991185c553ec05"}
Sep 16 18:51:26.214648 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:26.214610    2582 generic.go:356] "Generic (PLEG): container finished" podID="989138ce-8ff9-4abb-ad44-b07bfb5dcf59" containerID="db2538b76f2f08b7aad5461b805b2e8fb6a3cc06b5a6fe7057991185c553ec05" exitCode=0
Sep 16 18:51:26.215003 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:26.214700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zlp5j" event={"ID":"989138ce-8ff9-4abb-ad44-b07bfb5dcf59","Type":"ContainerDied","Data":"db2538b76f2f08b7aad5461b805b2e8fb6a3cc06b5a6fe7057991185c553ec05"}
Sep 16 18:51:31.240761 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:31.240722    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zlp5j" event={"ID":"989138ce-8ff9-4abb-ad44-b07bfb5dcf59","Type":"ContainerStarted","Data":"d19fc7b746e453fd25e462ba2c39a472dc3ec757a21886a76e99b545d8c3530d"}
Sep 16 18:51:31.261595 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:51:31.261534    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-zlp5j" podStartSLOduration=2.033243316 podStartE2EDuration="11.261514068s" totalImagesPullingTime="7.690114022s" totalInitContainerRuntime="1.53815673s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:31.260080432 +0000 UTC m=+686.447456223" watchObservedRunningTime="2026-09-16 18:51:31.261514068 +0000 UTC m=+686.448889859"
Sep 16 18:52:36.529014 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:36.528979    2582 generic.go:356] "Generic (PLEG): container finished" podID="989138ce-8ff9-4abb-ad44-b07bfb5dcf59" containerID="d19fc7b746e453fd25e462ba2c39a472dc3ec757a21886a76e99b545d8c3530d" exitCode=0
Sep 16 18:52:36.529403 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:36.529048    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zlp5j" event={"ID":"989138ce-8ff9-4abb-ad44-b07bfb5dcf59","Type":"ContainerDied","Data":"d19fc7b746e453fd25e462ba2c39a472dc3ec757a21886a76e99b545d8c3530d"}
Sep 16 18:52:37.655508 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:37.655489    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zlp5j"
Sep 16 18:52:37.700036 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:37.700009    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzdd9\" (UniqueName: \"kubernetes.io/projected/989138ce-8ff9-4abb-ad44-b07bfb5dcf59-kube-api-access-vzdd9\") pod \"989138ce-8ff9-4abb-ad44-b07bfb5dcf59\" (UID: \"989138ce-8ff9-4abb-ad44-b07bfb5dcf59\") "
Sep 16 18:52:37.700142 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:37.700045    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989138ce-8ff9-4abb-ad44-b07bfb5dcf59-model-cache\") pod \"989138ce-8ff9-4abb-ad44-b07bfb5dcf59\" (UID: \"989138ce-8ff9-4abb-ad44-b07bfb5dcf59\") "
Sep 16 18:52:37.702132 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:37.702102    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989138ce-8ff9-4abb-ad44-b07bfb5dcf59-kube-api-access-vzdd9" (OuterVolumeSpecName: "kube-api-access-vzdd9") pod "989138ce-8ff9-4abb-ad44-b07bfb5dcf59" (UID: "989138ce-8ff9-4abb-ad44-b07bfb5dcf59"). InnerVolumeSpecName "kube-api-access-vzdd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:37.718592 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:37.718567    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989138ce-8ff9-4abb-ad44-b07bfb5dcf59-model-cache" (OuterVolumeSpecName: "model-cache") pod "989138ce-8ff9-4abb-ad44-b07bfb5dcf59" (UID: "989138ce-8ff9-4abb-ad44-b07bfb5dcf59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:37.800611 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:37.800561    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzdd9\" (UniqueName: \"kubernetes.io/projected/989138ce-8ff9-4abb-ad44-b07bfb5dcf59-kube-api-access-vzdd9\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:52:37.800611 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:37.800582    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989138ce-8ff9-4abb-ad44-b07bfb5dcf59-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:52:38.539727 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:38.539701    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zlp5j"
Sep 16 18:52:38.539868 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:38.539729    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zlp5j" event={"ID":"989138ce-8ff9-4abb-ad44-b07bfb5dcf59","Type":"ContainerDied","Data":"b823e06a7bc5d9b9b9f628b037080a6dde4f580750bdb4508d53997edb6d8abc"}
Sep 16 18:52:38.539868 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:38.539767    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b823e06a7bc5d9b9b9f628b037080a6dde4f580750bdb4508d53997edb6d8abc"
Sep 16 18:52:46.448381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.448332    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-58mq6"]
Sep 16 18:52:46.448900 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.448866    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989138ce-8ff9-4abb-ad44-b07bfb5dcf59" containerName="s3-init"
Sep 16 18:52:46.448900 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.448887    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="989138ce-8ff9-4abb-ad44-b07bfb5dcf59" containerName="s3-init"
Sep 16 18:52:46.448900 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.448933    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989138ce-8ff9-4abb-ad44-b07bfb5dcf59" containerName="download-hf-model"
Sep 16 18:52:46.449131 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.448943    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="989138ce-8ff9-4abb-ad44-b07bfb5dcf59" containerName="download-hf-model"
Sep 16 18:52:46.449131 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.449073    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="989138ce-8ff9-4abb-ad44-b07bfb5dcf59" containerName="s3-init"
Sep 16 18:52:46.452785 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.452758    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-58mq6"
Sep 16 18:52:46.455500 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.455478    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:52:46.455659 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.455482    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:52:46.455659 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.455616    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2q5x2\""
Sep 16 18:52:46.457141 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.457123    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:52:46.457255 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.457164    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:52:46.464613 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.464589    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-58mq6"]
Sep 16 18:52:46.567137 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.567111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw29m\" (UniqueName: \"kubernetes.io/projected/65cc7e64-0c8d-474a-88e4-f6ca805cd69c-kube-api-access-jw29m\") pod \"s3-tls-init-custom-58mq6\" (UID: \"65cc7e64-0c8d-474a-88e4-f6ca805cd69c\") " pod="kserve/s3-tls-init-custom-58mq6"
Sep 16 18:52:46.567272 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.567170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65cc7e64-0c8d-474a-88e4-f6ca805cd69c-model-cache\") pod \"s3-tls-init-custom-58mq6\" (UID: \"65cc7e64-0c8d-474a-88e4-f6ca805cd69c\") " pod="kserve/s3-tls-init-custom-58mq6"
Sep 16 18:52:46.668208 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.668179    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65cc7e64-0c8d-474a-88e4-f6ca805cd69c-model-cache\") pod \"s3-tls-init-custom-58mq6\" (UID: \"65cc7e64-0c8d-474a-88e4-f6ca805cd69c\") " pod="kserve/s3-tls-init-custom-58mq6"
Sep 16 18:52:46.668327 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.668233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw29m\" (UniqueName: \"kubernetes.io/projected/65cc7e64-0c8d-474a-88e4-f6ca805cd69c-kube-api-access-jw29m\") pod \"s3-tls-init-custom-58mq6\" (UID: \"65cc7e64-0c8d-474a-88e4-f6ca805cd69c\") " pod="kserve/s3-tls-init-custom-58mq6"
Sep 16 18:52:46.668570 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.668549    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65cc7e64-0c8d-474a-88e4-f6ca805cd69c-model-cache\") pod \"s3-tls-init-custom-58mq6\" (UID: \"65cc7e64-0c8d-474a-88e4-f6ca805cd69c\") " pod="kserve/s3-tls-init-custom-58mq6"
Sep 16 18:52:46.678602 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.678570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw29m\" (UniqueName: \"kubernetes.io/projected/65cc7e64-0c8d-474a-88e4-f6ca805cd69c-kube-api-access-jw29m\") pod \"s3-tls-init-custom-58mq6\" (UID: \"65cc7e64-0c8d-474a-88e4-f6ca805cd69c\") " pod="kserve/s3-tls-init-custom-58mq6"
Sep 16 18:52:46.779192 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:46.779133    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-58mq6"
Sep 16 18:52:47.069669 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:47.069641    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-58mq6"]
Sep 16 18:52:47.072232 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:52:47.072201    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65cc7e64_0c8d_474a_88e4_f6ca805cd69c.slice/crio-4f9ebe92259056a5094b66bd01cd8bc60e7ca4631979d6282d1cd0bfd83e97fd WatchSource:0}: Error finding container 4f9ebe92259056a5094b66bd01cd8bc60e7ca4631979d6282d1cd0bfd83e97fd: Status 404 returned error can't find the container with id 4f9ebe92259056a5094b66bd01cd8bc60e7ca4631979d6282d1cd0bfd83e97fd
Sep 16 18:52:47.578235 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:47.578194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-58mq6" event={"ID":"65cc7e64-0c8d-474a-88e4-f6ca805cd69c","Type":"ContainerStarted","Data":"4376708e46ebd8d8d76d80525f4842271df84c3d0533a09dca23786e980690a9"}
Sep 16 18:52:47.578235 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:47.578241    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-58mq6" event={"ID":"65cc7e64-0c8d-474a-88e4-f6ca805cd69c","Type":"ContainerStarted","Data":"4f9ebe92259056a5094b66bd01cd8bc60e7ca4631979d6282d1cd0bfd83e97fd"}
Sep 16 18:52:48.583877 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:48.583839    2582 generic.go:356] "Generic (PLEG): container finished" podID="65cc7e64-0c8d-474a-88e4-f6ca805cd69c" containerID="4376708e46ebd8d8d76d80525f4842271df84c3d0533a09dca23786e980690a9" exitCode=0
Sep 16 18:52:48.584266 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:48.583935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-58mq6" event={"ID":"65cc7e64-0c8d-474a-88e4-f6ca805cd69c","Type":"ContainerDied","Data":"4376708e46ebd8d8d76d80525f4842271df84c3d0533a09dca23786e980690a9"}
Sep 16 18:52:49.590705 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:49.590674    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-58mq6" event={"ID":"65cc7e64-0c8d-474a-88e4-f6ca805cd69c","Type":"ContainerStarted","Data":"4be000392551c67ec7ad89bbae3a00f4dd82bfc9a08f1d11334d04011ba56b7d"}
Sep 16 18:52:49.610685 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:49.610631    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-58mq6" podStartSLOduration=2.312367357 podStartE2EDuration="3.610616584s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.298249227s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:49.609349164 +0000 UTC m=+764.796724952" watchObservedRunningTime="2026-09-16 18:52:49.610616584 +0000 UTC m=+764.797992371"
Sep 16 18:52:55.622738 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:55.622700    2582 generic.go:356] "Generic (PLEG): container finished" podID="65cc7e64-0c8d-474a-88e4-f6ca805cd69c" containerID="4be000392551c67ec7ad89bbae3a00f4dd82bfc9a08f1d11334d04011ba56b7d" exitCode=0
Sep 16 18:52:55.623223 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:55.622774    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-58mq6" event={"ID":"65cc7e64-0c8d-474a-88e4-f6ca805cd69c","Type":"ContainerDied","Data":"4be000392551c67ec7ad89bbae3a00f4dd82bfc9a08f1d11334d04011ba56b7d"}
Sep 16 18:52:56.780674 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:56.780645    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-58mq6"
Sep 16 18:52:56.850825 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:56.850802    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65cc7e64-0c8d-474a-88e4-f6ca805cd69c-model-cache\") pod \"65cc7e64-0c8d-474a-88e4-f6ca805cd69c\" (UID: \"65cc7e64-0c8d-474a-88e4-f6ca805cd69c\") "
Sep 16 18:52:56.850956 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:56.850869    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw29m\" (UniqueName: \"kubernetes.io/projected/65cc7e64-0c8d-474a-88e4-f6ca805cd69c-kube-api-access-jw29m\") pod \"65cc7e64-0c8d-474a-88e4-f6ca805cd69c\" (UID: \"65cc7e64-0c8d-474a-88e4-f6ca805cd69c\") "
Sep 16 18:52:56.853008 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:56.852986    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cc7e64-0c8d-474a-88e4-f6ca805cd69c-kube-api-access-jw29m" (OuterVolumeSpecName: "kube-api-access-jw29m") pod "65cc7e64-0c8d-474a-88e4-f6ca805cd69c" (UID: "65cc7e64-0c8d-474a-88e4-f6ca805cd69c"). InnerVolumeSpecName "kube-api-access-jw29m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:56.876537 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:56.876511    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cc7e64-0c8d-474a-88e4-f6ca805cd69c-model-cache" (OuterVolumeSpecName: "model-cache") pod "65cc7e64-0c8d-474a-88e4-f6ca805cd69c" (UID: "65cc7e64-0c8d-474a-88e4-f6ca805cd69c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:56.951538 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:56.951517    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw29m\" (UniqueName: \"kubernetes.io/projected/65cc7e64-0c8d-474a-88e4-f6ca805cd69c-kube-api-access-jw29m\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:52:56.951538 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:56.951537    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65cc7e64-0c8d-474a-88e4-f6ca805cd69c-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:52:57.636941 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:57.636860    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-58mq6"
Sep 16 18:52:57.636941 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:57.636868    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-58mq6" event={"ID":"65cc7e64-0c8d-474a-88e4-f6ca805cd69c","Type":"ContainerDied","Data":"4f9ebe92259056a5094b66bd01cd8bc60e7ca4631979d6282d1cd0bfd83e97fd"}
Sep 16 18:52:57.636941 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:52:57.636897    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f9ebe92259056a5094b66bd01cd8bc60e7ca4631979d6282d1cd0bfd83e97fd"
Sep 16 18:53:00.749780 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.749749    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-wnh9x"]
Sep 16 18:53:00.750178 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.750156    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65cc7e64-0c8d-474a-88e4-f6ca805cd69c" containerName="download-hf-model"
Sep 16 18:53:00.750178 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.750169    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="65cc7e64-0c8d-474a-88e4-f6ca805cd69c" containerName="download-hf-model"
Sep 16 18:53:00.750250 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.750179    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65cc7e64-0c8d-474a-88e4-f6ca805cd69c" containerName="s3-tls-init-custom"
Sep 16 18:53:00.750250 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.750189    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="65cc7e64-0c8d-474a-88e4-f6ca805cd69c" containerName="s3-tls-init-custom"
Sep 16 18:53:00.750318 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.750272    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="65cc7e64-0c8d-474a-88e4-f6ca805cd69c" containerName="s3-tls-init-custom"
Sep 16 18:53:00.753380 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.753363    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-wnh9x"
Sep 16 18:53:00.756213 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.756185    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:53:00.756213 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.756205    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 16 18:53:00.756407 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.756288    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2q5x2\""
Sep 16 18:53:00.757119 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.757101    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:53:00.757228 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.757128    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:53:00.763072 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.763048    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-wnh9x"]
Sep 16 18:53:00.876010 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.875990    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667cbff2-42d8-434d-8890-5ed1ae1e3d59-model-cache\") pod \"s3-tls-init-serving-wnh9x\" (UID: \"667cbff2-42d8-434d-8890-5ed1ae1e3d59\") " pod="kserve/s3-tls-init-serving-wnh9x"
Sep 16 18:53:00.876099 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.876037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl9hc\" (UniqueName: \"kubernetes.io/projected/667cbff2-42d8-434d-8890-5ed1ae1e3d59-kube-api-access-cl9hc\") pod \"s3-tls-init-serving-wnh9x\" (UID: \"667cbff2-42d8-434d-8890-5ed1ae1e3d59\") " pod="kserve/s3-tls-init-serving-wnh9x"
Sep 16 18:53:00.977197 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.977177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9hc\" (UniqueName: \"kubernetes.io/projected/667cbff2-42d8-434d-8890-5ed1ae1e3d59-kube-api-access-cl9hc\") pod \"s3-tls-init-serving-wnh9x\" (UID: \"667cbff2-42d8-434d-8890-5ed1ae1e3d59\") " pod="kserve/s3-tls-init-serving-wnh9x"
Sep 16 18:53:00.977303 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.977230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667cbff2-42d8-434d-8890-5ed1ae1e3d59-model-cache\") pod \"s3-tls-init-serving-wnh9x\" (UID: \"667cbff2-42d8-434d-8890-5ed1ae1e3d59\") " pod="kserve/s3-tls-init-serving-wnh9x"
Sep 16 18:53:00.977541 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.977525    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667cbff2-42d8-434d-8890-5ed1ae1e3d59-model-cache\") pod \"s3-tls-init-serving-wnh9x\" (UID: \"667cbff2-42d8-434d-8890-5ed1ae1e3d59\") " pod="kserve/s3-tls-init-serving-wnh9x"
Sep 16 18:53:00.988377 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:00.988343    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9hc\" (UniqueName: \"kubernetes.io/projected/667cbff2-42d8-434d-8890-5ed1ae1e3d59-kube-api-access-cl9hc\") pod \"s3-tls-init-serving-wnh9x\" (UID: \"667cbff2-42d8-434d-8890-5ed1ae1e3d59\") " pod="kserve/s3-tls-init-serving-wnh9x"
Sep 16 18:53:01.076178 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:01.076116    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-wnh9x"
Sep 16 18:53:01.348661 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:01.348638    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-wnh9x"]
Sep 16 18:53:01.350979 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:53:01.350944    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod667cbff2_42d8_434d_8890_5ed1ae1e3d59.slice/crio-5f148dc4605ea9a52c00bdb8f6808bcc35418c25ec1e839485f14e38df85ee9f WatchSource:0}: Error finding container 5f148dc4605ea9a52c00bdb8f6808bcc35418c25ec1e839485f14e38df85ee9f: Status 404 returned error can't find the container with id 5f148dc4605ea9a52c00bdb8f6808bcc35418c25ec1e839485f14e38df85ee9f
Sep 16 18:53:01.657538 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:01.657458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wnh9x" event={"ID":"667cbff2-42d8-434d-8890-5ed1ae1e3d59","Type":"ContainerStarted","Data":"2b6218d9f75956a2f0e8ac44bf07bd1d23a207d494013a71323ca68df3ee0494"}
Sep 16 18:53:01.657538 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:01.657505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wnh9x" event={"ID":"667cbff2-42d8-434d-8890-5ed1ae1e3d59","Type":"ContainerStarted","Data":"5f148dc4605ea9a52c00bdb8f6808bcc35418c25ec1e839485f14e38df85ee9f"}
Sep 16 18:53:03.667938 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:03.667885    2582 generic.go:356] "Generic (PLEG): container finished" podID="667cbff2-42d8-434d-8890-5ed1ae1e3d59" containerID="2b6218d9f75956a2f0e8ac44bf07bd1d23a207d494013a71323ca68df3ee0494" exitCode=0
Sep 16 18:53:03.667938 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:03.667936    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wnh9x" event={"ID":"667cbff2-42d8-434d-8890-5ed1ae1e3d59","Type":"ContainerDied","Data":"2b6218d9f75956a2f0e8ac44bf07bd1d23a207d494013a71323ca68df3ee0494"}
Sep 16 18:53:04.674399 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:04.674364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wnh9x" event={"ID":"667cbff2-42d8-434d-8890-5ed1ae1e3d59","Type":"ContainerStarted","Data":"918c7fefb9bc17c44cac82beb91bc04f13f8c458de7c6c0ca2bb7a502aeebdfb"}
Sep 16 18:53:04.693859 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:04.693811    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-wnh9x" podStartSLOduration=3.16956112 podStartE2EDuration="4.693796669s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.524235549s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:04.692551276 +0000 UTC m=+779.879927064" watchObservedRunningTime="2026-09-16 18:53:04.693796669 +0000 UTC m=+779.881172455"
Sep 16 18:53:09.701618 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:09.701583    2582 generic.go:356] "Generic (PLEG): container finished" podID="667cbff2-42d8-434d-8890-5ed1ae1e3d59" containerID="918c7fefb9bc17c44cac82beb91bc04f13f8c458de7c6c0ca2bb7a502aeebdfb" exitCode=0
Sep 16 18:53:09.702033 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:09.701658    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wnh9x" event={"ID":"667cbff2-42d8-434d-8890-5ed1ae1e3d59","Type":"ContainerDied","Data":"918c7fefb9bc17c44cac82beb91bc04f13f8c458de7c6c0ca2bb7a502aeebdfb"}
Sep 16 18:53:10.839293 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:10.839271    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-wnh9x"
Sep 16 18:53:10.961839 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:10.961780    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667cbff2-42d8-434d-8890-5ed1ae1e3d59-model-cache\") pod \"667cbff2-42d8-434d-8890-5ed1ae1e3d59\" (UID: \"667cbff2-42d8-434d-8890-5ed1ae1e3d59\") "
Sep 16 18:53:10.961962 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:10.961869    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl9hc\" (UniqueName: \"kubernetes.io/projected/667cbff2-42d8-434d-8890-5ed1ae1e3d59-kube-api-access-cl9hc\") pod \"667cbff2-42d8-434d-8890-5ed1ae1e3d59\" (UID: \"667cbff2-42d8-434d-8890-5ed1ae1e3d59\") "
Sep 16 18:53:10.964008 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:10.963982    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667cbff2-42d8-434d-8890-5ed1ae1e3d59-kube-api-access-cl9hc" (OuterVolumeSpecName: "kube-api-access-cl9hc") pod "667cbff2-42d8-434d-8890-5ed1ae1e3d59" (UID: "667cbff2-42d8-434d-8890-5ed1ae1e3d59"). InnerVolumeSpecName "kube-api-access-cl9hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:10.985806 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:10.985778    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667cbff2-42d8-434d-8890-5ed1ae1e3d59-model-cache" (OuterVolumeSpecName: "model-cache") pod "667cbff2-42d8-434d-8890-5ed1ae1e3d59" (UID: "667cbff2-42d8-434d-8890-5ed1ae1e3d59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:11.063415 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:11.063391    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl9hc\" (UniqueName: \"kubernetes.io/projected/667cbff2-42d8-434d-8890-5ed1ae1e3d59-kube-api-access-cl9hc\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:53:11.063492 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:11.063416    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667cbff2-42d8-434d-8890-5ed1ae1e3d59-model-cache\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:53:11.713628 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:11.713600    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-wnh9x"
Sep 16 18:53:11.713804 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:11.713630    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wnh9x" event={"ID":"667cbff2-42d8-434d-8890-5ed1ae1e3d59","Type":"ContainerDied","Data":"5f148dc4605ea9a52c00bdb8f6808bcc35418c25ec1e839485f14e38df85ee9f"}
Sep 16 18:53:11.713804 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:11.713676    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f148dc4605ea9a52c00bdb8f6808bcc35418c25ec1e839485f14e38df85ee9f"
Sep 16 18:53:24.068966 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.068937    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"]
Sep 16 18:53:24.069428 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.069277    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667cbff2-42d8-434d-8890-5ed1ae1e3d59" containerName="download-hf-model"
Sep 16 18:53:24.069428 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.069288    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="667cbff2-42d8-434d-8890-5ed1ae1e3d59" containerName="download-hf-model"
Sep 16 18:53:24.069428 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.069299    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667cbff2-42d8-434d-8890-5ed1ae1e3d59" containerName="s3-tls-init-serving"
Sep 16 18:53:24.069428 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.069305    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="667cbff2-42d8-434d-8890-5ed1ae1e3d59" containerName="s3-tls-init-serving"
Sep 16 18:53:24.069428 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.069355    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="667cbff2-42d8-434d-8890-5ed1ae1e3d59" containerName="s3-tls-init-serving"
Sep 16 18:53:24.073000 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.072980    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.075646 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.075621    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 16 18:53:24.075766 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.075702    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:53:24.075766 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.075740    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 18:53:24.075896 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.075764    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 16 18:53:24.075896 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.075788    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:53:24.075896 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.075846    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:53:24.075896 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.075888    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:53:24.085951 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.085928    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"]
Sep 16 18:53:24.261177 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.261152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.261314 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.261181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzmbb\" (UniqueName: \"kubernetes.io/projected/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kube-api-access-bzmbb\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.261314 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.261203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.261314 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.261224    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.261314 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.261239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.362511 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.362445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.362511 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.362477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmbb\" (UniqueName: \"kubernetes.io/projected/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kube-api-access-bzmbb\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.362511 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.362503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.362717 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.362622    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.362717 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.362662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.363071 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.363049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.363138 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.363089    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.363269 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.363253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.365035 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.365017    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.372844 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.372825    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmbb\" (UniqueName: \"kubernetes.io/projected/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kube-api-access-bzmbb\") pod \"isvc-sklearn-batcher-predictor-598fd877f-xqq2c\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.383642 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.383624    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:24.661058 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.661033    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"]
Sep 16 18:53:24.663719 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:53:24.663691    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5be4bc_640d_48cb_84e2_a8bc4df78e43.slice/crio-db28a2bceca5c02e98cc62a097dac8df601d827e51697c790537c8ebbe536274 WatchSource:0}: Error finding container db28a2bceca5c02e98cc62a097dac8df601d827e51697c790537c8ebbe536274: Status 404 returned error can't find the container with id db28a2bceca5c02e98cc62a097dac8df601d827e51697c790537c8ebbe536274
Sep 16 18:53:24.771589 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.771556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" event={"ID":"8d5be4bc-640d-48cb-84e2-a8bc4df78e43","Type":"ContainerStarted","Data":"e68d4c7df378d806812bb8aa9f588e81886f2498ce2fcb41afd7d387b63be97f"}
Sep 16 18:53:24.771707 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:24.771617    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" event={"ID":"8d5be4bc-640d-48cb-84e2-a8bc4df78e43","Type":"ContainerStarted","Data":"db28a2bceca5c02e98cc62a097dac8df601d827e51697c790537c8ebbe536274"}
Sep 16 18:53:28.791478 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:28.791402    2582 generic.go:356] "Generic (PLEG): container finished" podID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerID="e68d4c7df378d806812bb8aa9f588e81886f2498ce2fcb41afd7d387b63be97f" exitCode=0
Sep 16 18:53:28.791776 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:28.791476    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" event={"ID":"8d5be4bc-640d-48cb-84e2-a8bc4df78e43","Type":"ContainerDied","Data":"e68d4c7df378d806812bb8aa9f588e81886f2498ce2fcb41afd7d387b63be97f"}
Sep 16 18:53:41.864851 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:41.864824    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" event={"ID":"8d5be4bc-640d-48cb-84e2-a8bc4df78e43","Type":"ContainerStarted","Data":"80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33"}
Sep 16 18:53:44.886793 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:44.886754    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" event={"ID":"8d5be4bc-640d-48cb-84e2-a8bc4df78e43","Type":"ContainerStarted","Data":"52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e"}
Sep 16 18:53:46.898079 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:46.898045    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" event={"ID":"8d5be4bc-640d-48cb-84e2-a8bc4df78e43","Type":"ContainerStarted","Data":"fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f"}
Sep 16 18:53:46.898510 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:46.898269    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:46.898510 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:46.898446    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:46.899661 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:46.899623    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 18:53:46.922106 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:46.922057    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podStartSLOduration=1.894258451 podStartE2EDuration="22.922040713s" totalImagesPullingTime="17.241876697s" totalInitContainerRuntime="3.785905565s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:46.920240816 +0000 UTC m=+822.107616605" watchObservedRunningTime="2026-09-16 18:53:46.922040713 +0000 UTC m=+822.109416500"
Sep 16 18:53:47.902667 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:47.902630    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:47.903062 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:47.902678    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 18:53:47.903887 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:47.903854    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:48.907468 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:48.907429    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 18:53:48.907977 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:48.907944    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:53.912863 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:53.912832    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:53:53.913428 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:53.913392    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 18:53:53.913806 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:53:53.913780    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:03.914002 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:54:03.913962    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 18:54:03.914729 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:54:03.914700    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:13.916803 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:54:13.916757    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 18:54:13.917310 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:54:13.917216    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:23.914045 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:54:23.914003    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 18:54:23.914486 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:54:23.914460    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:33.914218 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:54:33.914177    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 18:54:33.914673 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:54:33.914587    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:43.913417 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:54:43.913380    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 18:54:43.913938 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:54:43.913801    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:53.914114 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:54:53.914078    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:54:53.914669 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:54:53.914362    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:55:05.249706 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:05.249625    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 18:55:05.250730 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:05.250704    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 18:55:09.089482 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.089452    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"]
Sep 16 18:55:09.089877 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.089692    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" containerID="cri-o://80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33" gracePeriod=30
Sep 16 18:55:09.089877 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.089721    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent" containerID="cri-o://fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f" gracePeriod=30
Sep 16 18:55:09.089877 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.089735    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kube-rbac-proxy" containerID="cri-o://52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e" gracePeriod=30
Sep 16 18:55:09.249717 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.249691    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"]
Sep 16 18:55:09.253147 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.253130    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.255348 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.255327    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 16 18:55:09.255824 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.255803    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 16 18:55:09.271467 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.271447    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"]
Sep 16 18:55:09.282353 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.282329    2582 generic.go:356] "Generic (PLEG): container finished" podID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerID="52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e" exitCode=2
Sep 16 18:55:09.282429 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.282364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" event={"ID":"8d5be4bc-640d-48cb-84e2-a8bc4df78e43","Type":"ContainerDied","Data":"52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e"}
Sep 16 18:55:09.362613 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.362550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7qln\" (UniqueName: \"kubernetes.io/projected/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kube-api-access-g7qln\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.362717 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.362648    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.362717 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.362692    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.362803 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.362768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.362851 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.362812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.463557 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.463533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.463642 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.463575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.463642 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.463594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qln\" (UniqueName: \"kubernetes.io/projected/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kube-api-access-g7qln\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.463642 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.463627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.463760 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.463653    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.464146 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.464124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.464247 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.464230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.464344 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.464328    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.466318 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.466299    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.473700 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.473676    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7qln\" (UniqueName: \"kubernetes.io/projected/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kube-api-access-g7qln\") pod \"isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.563147 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.563126    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:09.841760 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:09.841739    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"]
Sep 16 18:55:09.843976 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:55:09.843950    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459b94c6_cd7b_48ce_b1f7_63ea689cb5b1.slice/crio-820100ac5dd68fa38de409b4fc93c03f63c205efa3c313d319ad6dd153da13b5 WatchSource:0}: Error finding container 820100ac5dd68fa38de409b4fc93c03f63c205efa3c313d319ad6dd153da13b5: Status 404 returned error can't find the container with id 820100ac5dd68fa38de409b4fc93c03f63c205efa3c313d319ad6dd153da13b5
Sep 16 18:55:10.296306 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:10.296274    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" event={"ID":"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1","Type":"ContainerStarted","Data":"342f93f11ad84962c022313086c0e2b8ed6e97ea631e554201e5fb751f90aea8"}
Sep 16 18:55:10.296777 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:10.296333    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" event={"ID":"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1","Type":"ContainerStarted","Data":"820100ac5dd68fa38de409b4fc93c03f63c205efa3c313d319ad6dd153da13b5"}
Sep 16 18:55:13.311943 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:13.311882    2582 generic.go:356] "Generic (PLEG): container finished" podID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerID="80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33" exitCode=0
Sep 16 18:55:13.312237 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:13.311960    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" event={"ID":"8d5be4bc-640d-48cb-84e2-a8bc4df78e43","Type":"ContainerDied","Data":"80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33"}
Sep 16 18:55:13.908519 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:13.908442    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 16 18:55:13.913865 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:13.913832    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 18:55:13.914326 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:13.914298    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:14.320002 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:14.319975    2582 generic.go:356] "Generic (PLEG): container finished" podID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerID="342f93f11ad84962c022313086c0e2b8ed6e97ea631e554201e5fb751f90aea8" exitCode=0
Sep 16 18:55:14.320443 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:14.320013    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" event={"ID":"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1","Type":"ContainerDied","Data":"342f93f11ad84962c022313086c0e2b8ed6e97ea631e554201e5fb751f90aea8"}
Sep 16 18:55:15.327553 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:15.327517    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" event={"ID":"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1","Type":"ContainerStarted","Data":"6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5"}
Sep 16 18:55:15.328062 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:15.327595    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" event={"ID":"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1","Type":"ContainerStarted","Data":"c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f"}
Sep 16 18:55:15.328062 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:15.327634    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" event={"ID":"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1","Type":"ContainerStarted","Data":"0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3"}
Sep 16 18:55:15.328062 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:15.327956    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:15.328062 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:15.328018    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:15.328062 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:15.328042    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:15.329483 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:15.329457    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Sep 16 18:55:15.330176 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:15.330152    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:15.350748 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:15.350705    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podStartSLOduration=2.629265419 podStartE2EDuration="6.350692076s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.721426657s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:15.349293825 +0000 UTC m=+910.536669626" watchObservedRunningTime="2026-09-16 18:55:15.350692076 +0000 UTC m=+910.538067863"
Sep 16 18:55:16.333098 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:16.333062    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Sep 16 18:55:16.333552 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:16.333387    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:18.908295 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:18.908257    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 16 18:55:21.337527 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:21.337499    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:55:21.338135 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:21.338102    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Sep 16 18:55:21.338379 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:21.338355    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:23.908285 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:23.908238    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 16 18:55:23.908638 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:23.908367    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:55:23.913286 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:23.913252    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 18:55:23.913631 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:23.913605    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:28.907833 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:28.907791    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 16 18:55:31.338780 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:31.338744    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Sep 16 18:55:31.339357 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:31.339320    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:33.907709 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:33.907669    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 16 18:55:33.914141 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:33.914112    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 16 18:55:33.914273 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:33.914257    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:55:33.914500 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:33.914465    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:33.914603 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:33.914590    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:55:38.907501 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:38.907460    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 16 18:55:39.242193 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.242172    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:55:39.396135 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.396108    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kserve-provision-location\") pod \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") "
Sep 16 18:55:39.396297 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.396145    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzmbb\" (UniqueName: \"kubernetes.io/projected/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kube-api-access-bzmbb\") pod \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") "
Sep 16 18:55:39.396297 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.396178    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kube-rbac-proxy-sar-config\") pod \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") "
Sep 16 18:55:39.396297 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.396207    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-proxy-tls\") pod \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") "
Sep 16 18:55:39.396297 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.396267    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-cabundle-cert\") pod \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\" (UID: \"8d5be4bc-640d-48cb-84e2-a8bc4df78e43\") "
Sep 16 18:55:39.396721 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.396469    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d5be4bc-640d-48cb-84e2-a8bc4df78e43" (UID: "8d5be4bc-640d-48cb-84e2-a8bc4df78e43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:39.396721 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.396543    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8d5be4bc-640d-48cb-84e2-a8bc4df78e43" (UID: "8d5be4bc-640d-48cb-84e2-a8bc4df78e43"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:39.396815 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.396761    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d5be4bc-640d-48cb-84e2-a8bc4df78e43" (UID: "8d5be4bc-640d-48cb-84e2-a8bc4df78e43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:39.398951 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.398886    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kube-api-access-bzmbb" (OuterVolumeSpecName: "kube-api-access-bzmbb") pod "8d5be4bc-640d-48cb-84e2-a8bc4df78e43" (UID: "8d5be4bc-640d-48cb-84e2-a8bc4df78e43"). InnerVolumeSpecName "kube-api-access-bzmbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:39.398951 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.398892    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d5be4bc-640d-48cb-84e2-a8bc4df78e43" (UID: "8d5be4bc-640d-48cb-84e2-a8bc4df78e43"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:39.441631 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.441604    2582 generic.go:356] "Generic (PLEG): container finished" podID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerID="fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f" exitCode=0
Sep 16 18:55:39.441759 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.441649    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" event={"ID":"8d5be4bc-640d-48cb-84e2-a8bc4df78e43","Type":"ContainerDied","Data":"fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f"}
Sep 16 18:55:39.441759 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.441682    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"
Sep 16 18:55:39.441759 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.441734    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c" event={"ID":"8d5be4bc-640d-48cb-84e2-a8bc4df78e43","Type":"ContainerDied","Data":"db28a2bceca5c02e98cc62a097dac8df601d827e51697c790537c8ebbe536274"}
Sep 16 18:55:39.441986 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.441780    2582 scope.go:117] "RemoveContainer" containerID="fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f"
Sep 16 18:55:39.451446 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.451431    2582 scope.go:117] "RemoveContainer" containerID="52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e"
Sep 16 18:55:39.459295 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.459279    2582 scope.go:117] "RemoveContainer" containerID="80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33"
Sep 16 18:55:39.466447 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.466431    2582 scope.go:117] "RemoveContainer" containerID="e68d4c7df378d806812bb8aa9f588e81886f2498ce2fcb41afd7d387b63be97f"
Sep 16 18:55:39.470474 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.470454    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"]
Sep 16 18:55:39.475200 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.475155    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c"]
Sep 16 18:55:39.477277 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.477252    2582 scope.go:117] "RemoveContainer" containerID="fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f"
Sep 16 18:55:39.477558 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:55:39.477541    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f\": container with ID starting with fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f not found: ID does not exist" containerID="fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f"
Sep 16 18:55:39.477621 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.477567    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f"} err="failed to get container status \"fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f\": rpc error: code = NotFound desc = could not find container \"fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f\": container with ID starting with fe8935d92a3e89ba4c1f31122bcba44a1d12fb69bf3d49f982701c160495b87f not found: ID does not exist"
Sep 16 18:55:39.477621 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.477588    2582 scope.go:117] "RemoveContainer" containerID="52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e"
Sep 16 18:55:39.477896 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:55:39.477871    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e\": container with ID starting with 52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e not found: ID does not exist" containerID="52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e"
Sep 16 18:55:39.477986 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.477922    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e"} err="failed to get container status \"52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e\": rpc error: code = NotFound desc = could not find container \"52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e\": container with ID starting with 52bfe1df2da95fcf41a1702ac54a95df8b26b25512fba941ddfe08cab8ae361e not found: ID does not exist"
Sep 16 18:55:39.477986 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.477944    2582 scope.go:117] "RemoveContainer" containerID="80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33"
Sep 16 18:55:39.478187 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:55:39.478169    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33\": container with ID starting with 80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33 not found: ID does not exist" containerID="80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33"
Sep 16 18:55:39.478237 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.478191    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33"} err="failed to get container status \"80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33\": rpc error: code = NotFound desc = could not find container \"80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33\": container with ID starting with 80593076cb1e132d20cb57fc91477231e51764cd2d1cdfa30997b590a5ed8c33 not found: ID does not exist"
Sep 16 18:55:39.478237 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.478207    2582 scope.go:117] "RemoveContainer" containerID="e68d4c7df378d806812bb8aa9f588e81886f2498ce2fcb41afd7d387b63be97f"
Sep 16 18:55:39.478487 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:55:39.478468    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68d4c7df378d806812bb8aa9f588e81886f2498ce2fcb41afd7d387b63be97f\": container with ID starting with e68d4c7df378d806812bb8aa9f588e81886f2498ce2fcb41afd7d387b63be97f not found: ID does not exist" containerID="e68d4c7df378d806812bb8aa9f588e81886f2498ce2fcb41afd7d387b63be97f"
Sep 16 18:55:39.478579 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.478487    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68d4c7df378d806812bb8aa9f588e81886f2498ce2fcb41afd7d387b63be97f"} err="failed to get container status \"e68d4c7df378d806812bb8aa9f588e81886f2498ce2fcb41afd7d387b63be97f\": rpc error: code = NotFound desc = could not find container \"e68d4c7df378d806812bb8aa9f588e81886f2498ce2fcb41afd7d387b63be97f\": container with ID starting with e68d4c7df378d806812bb8aa9f588e81886f2498ce2fcb41afd7d387b63be97f not found: ID does not exist"
Sep 16 18:55:39.496843 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.496822    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.496954 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.496844    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.496954 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.496857    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.496954 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.496864    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.496954 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:39.496872    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzmbb\" (UniqueName: \"kubernetes.io/projected/8d5be4bc-640d-48cb-84e2-a8bc4df78e43-kube-api-access-bzmbb\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:55:41.296360 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:41.296325    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" path="/var/lib/kubelet/pods/8d5be4bc-640d-48cb-84e2-a8bc4df78e43/volumes"
Sep 16 18:55:41.337945 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:41.337892    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Sep 16 18:55:41.338301 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:41.338279    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:51.338652 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:51.338609    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Sep 16 18:55:51.339161 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:55:51.339089    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:01.338787 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:01.338748    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Sep 16 18:56:01.339302 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:01.339238    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:11.337994 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:11.337955    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Sep 16 18:56:11.338495 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:11.338468    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:21.338786 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:21.338744    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Sep 16 18:56:21.339357 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:21.339216    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:31.339232 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:31.339124    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:56:31.339857 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:31.339244    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:56:44.294954 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:44.294923    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"]
Sep 16 18:56:44.295467 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:44.295209    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" containerID="cri-o://0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3" gracePeriod=30
Sep 16 18:56:44.295467 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:44.295240    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" containerID="cri-o://6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5" gracePeriod=30
Sep 16 18:56:44.295467 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:44.295381    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kube-rbac-proxy" containerID="cri-o://c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f" gracePeriod=30
Sep 16 18:56:44.738670 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:44.738642    2582 generic.go:356] "Generic (PLEG): container finished" podID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerID="c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f" exitCode=2
Sep 16 18:56:44.738802 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:44.738714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" event={"ID":"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1","Type":"ContainerDied","Data":"c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f"}
Sep 16 18:56:46.333635 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:46.333601    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 16 18:56:48.763216 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:48.763177    2582 generic.go:356] "Generic (PLEG): container finished" podID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerID="0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3" exitCode=0
Sep 16 18:56:48.763602 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:48.763250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" event={"ID":"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1","Type":"ContainerDied","Data":"0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3"}
Sep 16 18:56:51.333678 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:51.333634    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 16 18:56:51.338047 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:51.338014    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Sep 16 18:56:51.338411 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:51.338384    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:56.333198 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:56.333147    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 16 18:56:56.333646 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:56:56.333315    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:57:01.333255 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:01.333211    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 16 18:57:01.338634 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:01.338597    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Sep 16 18:57:01.338991 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:01.338960    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:04.598849 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.598809    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"]
Sep 16 18:57:04.599388 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.599368    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kube-rbac-proxy"
Sep 16 18:57:04.599475 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.599391    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kube-rbac-proxy"
Sep 16 18:57:04.599475 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.599405    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent"
Sep 16 18:57:04.599475 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.599421    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent"
Sep 16 18:57:04.599475 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.599435    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="storage-initializer"
Sep 16 18:57:04.599475 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.599444    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="storage-initializer"
Sep 16 18:57:04.599475 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.599453    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container"
Sep 16 18:57:04.599475 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.599461    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container"
Sep 16 18:57:04.599831 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.599582    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kserve-container"
Sep 16 18:57:04.599831 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.599600    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="agent"
Sep 16 18:57:04.599831 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.599610    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d5be4bc-640d-48cb-84e2-a8bc4df78e43" containerName="kube-rbac-proxy"
Sep 16 18:57:04.603173 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.603151    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.610362 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.610344    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 16 18:57:04.610437 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.610356    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 16 18:57:04.641029 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.641001    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"]
Sep 16 18:57:04.734760 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.734727    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48145d0a-85c8-47de-b218-ab3bbafead09-proxy-tls\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.734871 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.734770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48145d0a-85c8-47de-b218-ab3bbafead09-kserve-provision-location\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.734871 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.734790    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48145d0a-85c8-47de-b218-ab3bbafead09-cabundle-cert\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.735033 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.734928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfshj\" (UniqueName: \"kubernetes.io/projected/48145d0a-85c8-47de-b218-ab3bbafead09-kube-api-access-vfshj\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.735033 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.734977    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48145d0a-85c8-47de-b218-ab3bbafead09-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.836273 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.836243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfshj\" (UniqueName: \"kubernetes.io/projected/48145d0a-85c8-47de-b218-ab3bbafead09-kube-api-access-vfshj\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.836385 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.836278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48145d0a-85c8-47de-b218-ab3bbafead09-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.836449 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.836405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48145d0a-85c8-47de-b218-ab3bbafead09-proxy-tls\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.836449 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.836435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48145d0a-85c8-47de-b218-ab3bbafead09-kserve-provision-location\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.836566 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.836463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48145d0a-85c8-47de-b218-ab3bbafead09-cabundle-cert\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.836776 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.836756    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48145d0a-85c8-47de-b218-ab3bbafead09-kserve-provision-location\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.836964 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.836948    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48145d0a-85c8-47de-b218-ab3bbafead09-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.837010 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.836999    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48145d0a-85c8-47de-b218-ab3bbafead09-cabundle-cert\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.838897 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.838880    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48145d0a-85c8-47de-b218-ab3bbafead09-proxy-tls\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.852681 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.852637    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfshj\" (UniqueName: \"kubernetes.io/projected/48145d0a-85c8-47de-b218-ab3bbafead09-kube-api-access-vfshj\") pod \"isvc-logger-predictor-5d8c64c8b4-9mcc7\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:04.913522 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:04.913502    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:05.194798 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:05.194771    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"]
Sep 16 18:57:05.195795 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:57:05.195770    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48145d0a_85c8_47de_b218_ab3bbafead09.slice/crio-35b7ce2b6152bfcd89fdb0f7be7decdd5d2a439cfd746e6de2947285fcf6fffe WatchSource:0}: Error finding container 35b7ce2b6152bfcd89fdb0f7be7decdd5d2a439cfd746e6de2947285fcf6fffe: Status 404 returned error can't find the container with id 35b7ce2b6152bfcd89fdb0f7be7decdd5d2a439cfd746e6de2947285fcf6fffe
Sep 16 18:57:05.840681 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:05.840638    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" event={"ID":"48145d0a-85c8-47de-b218-ab3bbafead09","Type":"ContainerStarted","Data":"06c3204e51aaf2c1bc7f75d8a353bb8c67730a1d3ce97d9dc6bdd8ff3a44fb64"}
Sep 16 18:57:05.841173 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:05.840739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" event={"ID":"48145d0a-85c8-47de-b218-ab3bbafead09","Type":"ContainerStarted","Data":"35b7ce2b6152bfcd89fdb0f7be7decdd5d2a439cfd746e6de2947285fcf6fffe"}
Sep 16 18:57:06.333623 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:06.333567    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 16 18:57:09.865664 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:09.865626    2582 generic.go:356] "Generic (PLEG): container finished" podID="48145d0a-85c8-47de-b218-ab3bbafead09" containerID="06c3204e51aaf2c1bc7f75d8a353bb8c67730a1d3ce97d9dc6bdd8ff3a44fb64" exitCode=0
Sep 16 18:57:09.866115 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:09.865689    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" event={"ID":"48145d0a-85c8-47de-b218-ab3bbafead09","Type":"ContainerDied","Data":"06c3204e51aaf2c1bc7f75d8a353bb8c67730a1d3ce97d9dc6bdd8ff3a44fb64"}
Sep 16 18:57:10.879805 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:10.879772    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" event={"ID":"48145d0a-85c8-47de-b218-ab3bbafead09","Type":"ContainerStarted","Data":"679db500ad5e8e6c9f41e1356a5d246475c68b991c7cb182acb3bcd131c2e79c"}
Sep 16 18:57:10.880222 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:10.879871    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" event={"ID":"48145d0a-85c8-47de-b218-ab3bbafead09","Type":"ContainerStarted","Data":"d9b9806050c889463fba6d7b672dd24bbab83a7a41d351372a82947c9ef669a7"}
Sep 16 18:57:10.880222 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:10.879924    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" event={"ID":"48145d0a-85c8-47de-b218-ab3bbafead09","Type":"ContainerStarted","Data":"0c6394d3700e2f06782eb81bf2b49a589b43635943013dd5e25b013be17d7dfc"}
Sep 16 18:57:10.880340 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:10.880231    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:10.880408 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:10.880391    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:10.881518 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:10.881495    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:57:10.911495 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:10.911455    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podStartSLOduration=3.240294694 podStartE2EDuration="6.911442349s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.671147655s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:10.90979702 +0000 UTC m=+1026.097172842" watchObservedRunningTime="2026-09-16 18:57:10.911442349 +0000 UTC m=+1026.098818114"
Sep 16 18:57:11.333668 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:11.333633    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 16 18:57:11.337883 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:11.337857    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Sep 16 18:57:11.338034 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:11.337997    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:57:11.338304 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:11.338282    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:11.338402 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:11.338388    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:57:11.885450 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:11.885416    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:11.885928 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:11.885482    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:57:11.886521 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:11.886495    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:12.890876 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:12.890836    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:57:12.891304 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:12.891188    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:14.444138 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.444109    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:57:14.512984 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.512965    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7qln\" (UniqueName: \"kubernetes.io/projected/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kube-api-access-g7qln\") pod \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") "
Sep 16 18:57:14.513107 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.512998    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-cabundle-cert\") pod \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") "
Sep 16 18:57:14.513107 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.513053    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kserve-provision-location\") pod \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") "
Sep 16 18:57:14.513107 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.513081    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kube-rbac-proxy-sar-config\") pod \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") "
Sep 16 18:57:14.513244 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.513119    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-proxy-tls\") pod \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\" (UID: \"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1\") "
Sep 16 18:57:14.513411 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.513376    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" (UID: "459b94c6-cd7b-48ce-b1f7-63ea689cb5b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:14.513483 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.513401    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" (UID: "459b94c6-cd7b-48ce-b1f7-63ea689cb5b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:14.513483 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.513422    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" (UID: "459b94c6-cd7b-48ce-b1f7-63ea689cb5b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:14.515253 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.515232    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kube-api-access-g7qln" (OuterVolumeSpecName: "kube-api-access-g7qln") pod "459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" (UID: "459b94c6-cd7b-48ce-b1f7-63ea689cb5b1"). InnerVolumeSpecName "kube-api-access-g7qln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:14.515345 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.515330    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" (UID: "459b94c6-cd7b-48ce-b1f7-63ea689cb5b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:14.614634 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.614581    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7qln\" (UniqueName: \"kubernetes.io/projected/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kube-api-access-g7qln\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:57:14.614634 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.614601    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:57:14.614634 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.614609    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:57:14.614634 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.614618    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:57:14.614634 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.614625    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:57:14.902596 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.902514    2582 generic.go:356] "Generic (PLEG): container finished" podID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerID="6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5" exitCode=0
Sep 16 18:57:14.902596 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.902557    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" event={"ID":"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1","Type":"ContainerDied","Data":"6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5"}
Sep 16 18:57:14.902596 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.902583    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"
Sep 16 18:57:14.902857 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.902623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz" event={"ID":"459b94c6-cd7b-48ce-b1f7-63ea689cb5b1","Type":"ContainerDied","Data":"820100ac5dd68fa38de409b4fc93c03f63c205efa3c313d319ad6dd153da13b5"}
Sep 16 18:57:14.902857 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.902660    2582 scope.go:117] "RemoveContainer" containerID="6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5"
Sep 16 18:57:14.912295 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.912279    2582 scope.go:117] "RemoveContainer" containerID="c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f"
Sep 16 18:57:14.920852 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.920834    2582 scope.go:117] "RemoveContainer" containerID="0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3"
Sep 16 18:57:14.928547 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.928532    2582 scope.go:117] "RemoveContainer" containerID="342f93f11ad84962c022313086c0e2b8ed6e97ea631e554201e5fb751f90aea8"
Sep 16 18:57:14.936061 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.936037    2582 scope.go:117] "RemoveContainer" containerID="6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5"
Sep 16 18:57:14.936346 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:57:14.936328    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5\": container with ID starting with 6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5 not found: ID does not exist" containerID="6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5"
Sep 16 18:57:14.936410 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.936354    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5"} err="failed to get container status \"6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5\": rpc error: code = NotFound desc = could not find container \"6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5\": container with ID starting with 6d61d31680bab0e07e3a43664b4b6374d49765a92ee7ba38d21a910617012fa5 not found: ID does not exist"
Sep 16 18:57:14.936410 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.936368    2582 scope.go:117] "RemoveContainer" containerID="c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f"
Sep 16 18:57:14.936603 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:57:14.936587    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f\": container with ID starting with c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f not found: ID does not exist" containerID="c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f"
Sep 16 18:57:14.936651 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.936611    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f"} err="failed to get container status \"c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f\": rpc error: code = NotFound desc = could not find container \"c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f\": container with ID starting with c59d4c6d846be3eaf019459895538dd0c2dc1210203d72b9f919cbbbf252569f not found: ID does not exist"
Sep 16 18:57:14.936651 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.936625    2582 scope.go:117] "RemoveContainer" containerID="0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3"
Sep 16 18:57:14.936840 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:57:14.936827    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3\": container with ID starting with 0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3 not found: ID does not exist" containerID="0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3"
Sep 16 18:57:14.936885 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.936843    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3"} err="failed to get container status \"0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3\": rpc error: code = NotFound desc = could not find container \"0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3\": container with ID starting with 0bb7d15c6599657038ea515d2d93ecfc78446c47ec916dad2ceaa8155278b7a3 not found: ID does not exist"
Sep 16 18:57:14.936885 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.936854    2582 scope.go:117] "RemoveContainer" containerID="342f93f11ad84962c022313086c0e2b8ed6e97ea631e554201e5fb751f90aea8"
Sep 16 18:57:14.937153 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:57:14.937137    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342f93f11ad84962c022313086c0e2b8ed6e97ea631e554201e5fb751f90aea8\": container with ID starting with 342f93f11ad84962c022313086c0e2b8ed6e97ea631e554201e5fb751f90aea8 not found: ID does not exist" containerID="342f93f11ad84962c022313086c0e2b8ed6e97ea631e554201e5fb751f90aea8"
Sep 16 18:57:14.937195 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.937160    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342f93f11ad84962c022313086c0e2b8ed6e97ea631e554201e5fb751f90aea8"} err="failed to get container status \"342f93f11ad84962c022313086c0e2b8ed6e97ea631e554201e5fb751f90aea8\": rpc error: code = NotFound desc = could not find container \"342f93f11ad84962c022313086c0e2b8ed6e97ea631e554201e5fb751f90aea8\": container with ID starting with 342f93f11ad84962c022313086c0e2b8ed6e97ea631e554201e5fb751f90aea8 not found: ID does not exist"
Sep 16 18:57:14.937601 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.937584    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"]
Sep 16 18:57:14.951094 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:14.951073    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz"]
Sep 16 18:57:15.296322 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:15.296298    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" path="/var/lib/kubelet/pods/459b94c6-cd7b-48ce-b1f7-63ea689cb5b1/volumes"
Sep 16 18:57:17.895115 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:17.895090    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:57:17.895644 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:17.895616    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:57:17.895932 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:17.895882    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:27.896426 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:27.896386    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:57:27.897013 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:27.896895    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:37.896313 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:37.896272    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:57:37.896853 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:37.896825    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:47.895714 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:47.895670    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:57:47.896256 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:47.895954    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:57.895578 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:57.895490    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:57:57.895967 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:57:57.895941    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:07.896471 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:07.896432    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:58:07.896941 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:07.896874    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:17.896144 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:17.896098    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:58:17.896650 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:17.896601    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:22.292113 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:22.292086    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:58:22.292569 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:22.292355    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:58:29.861136 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:29.861102    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"]
Sep 16 18:58:29.861536 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:29.861374    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" containerID="cri-o://0c6394d3700e2f06782eb81bf2b49a589b43635943013dd5e25b013be17d7dfc" gracePeriod=30
Sep 16 18:58:29.861536 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:29.861393    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kube-rbac-proxy" containerID="cri-o://d9b9806050c889463fba6d7b672dd24bbab83a7a41d351372a82947c9ef669a7" gracePeriod=30
Sep 16 18:58:29.861536 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:29.861397    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" containerID="cri-o://679db500ad5e8e6c9f41e1356a5d246475c68b991c7cb182acb3bcd131c2e79c" gracePeriod=30
Sep 16 18:58:30.257516 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:30.257485    2582 generic.go:356] "Generic (PLEG): container finished" podID="48145d0a-85c8-47de-b218-ab3bbafead09" containerID="d9b9806050c889463fba6d7b672dd24bbab83a7a41d351372a82947c9ef669a7" exitCode=2
Sep 16 18:58:30.257646 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:30.257559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" event={"ID":"48145d0a-85c8-47de-b218-ab3bbafead09","Type":"ContainerDied","Data":"d9b9806050c889463fba6d7b672dd24bbab83a7a41d351372a82947c9ef669a7"}
Sep 16 18:58:32.292434 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:32.292397    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:58:32.293183 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:32.293156    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:32.891653 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:32.891618    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 16 18:58:34.277828 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:34.277798    2582 generic.go:356] "Generic (PLEG): container finished" podID="48145d0a-85c8-47de-b218-ab3bbafead09" containerID="0c6394d3700e2f06782eb81bf2b49a589b43635943013dd5e25b013be17d7dfc" exitCode=0
Sep 16 18:58:34.278221 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:34.277867    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" event={"ID":"48145d0a-85c8-47de-b218-ab3bbafead09","Type":"ContainerDied","Data":"0c6394d3700e2f06782eb81bf2b49a589b43635943013dd5e25b013be17d7dfc"}
Sep 16 18:58:37.890876 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:37.890832    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 16 18:58:42.292537 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:42.292484    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:58:42.293072 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:42.292939    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:42.891693 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:42.891650    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 16 18:58:42.891901 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:42.891791    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:58:47.891256 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:47.891208    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 16 18:58:50.201054 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.201018    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"]
Sep 16 18:58:50.201549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.201371    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent"
Sep 16 18:58:50.201549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.201382    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent"
Sep 16 18:58:50.201549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.201401    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container"
Sep 16 18:58:50.201549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.201407    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container"
Sep 16 18:58:50.201549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.201419    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="storage-initializer"
Sep 16 18:58:50.201549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.201424    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="storage-initializer"
Sep 16 18:58:50.201549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.201432    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kube-rbac-proxy"
Sep 16 18:58:50.201549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.201437    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kube-rbac-proxy"
Sep 16 18:58:50.201549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.201482    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kube-rbac-proxy"
Sep 16 18:58:50.201549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.201490    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="agent"
Sep 16 18:58:50.201549 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.201497    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="459b94c6-cd7b-48ce-b1f7-63ea689cb5b1" containerName="kserve-container"
Sep 16 18:58:50.204952 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.204932    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.210686 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.210667    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 16 18:58:50.211389 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.211348    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 16 18:58:50.226491 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.226466    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"]
Sep 16 18:58:50.334456 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.334428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.334560 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.334472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cx75\" (UniqueName: \"kubernetes.io/projected/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kube-api-access-7cx75\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.334560 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.334503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e4ff34-7b79-4e55-a3a8-29a06758e550-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.334560 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.334548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e4ff34-7b79-4e55-a3a8-29a06758e550-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.334672 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.334565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.435133 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.435099    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cx75\" (UniqueName: \"kubernetes.io/projected/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kube-api-access-7cx75\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.435264 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.435151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e4ff34-7b79-4e55-a3a8-29a06758e550-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.435264 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.435169    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e4ff34-7b79-4e55-a3a8-29a06758e550-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.435264 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.435186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.435264 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.435224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.435498 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:58:50.435284    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 16 18:58:50.435498 ip-10-0-129-50 kubenswrapper[2582]: E0916 18:58:50.435358    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1e4ff34-7b79-4e55-a3a8-29a06758e550-proxy-tls podName:d1e4ff34-7b79-4e55-a3a8-29a06758e550 nodeName:}" failed. No retries permitted until 2026-09-16 18:58:50.935337365 +0000 UTC m=+1126.122713133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d1e4ff34-7b79-4e55-a3a8-29a06758e550-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" (UID: "d1e4ff34-7b79-4e55-a3a8-29a06758e550") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 16 18:58:50.435679 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.435655    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.435895 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.435874    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.436005 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.435890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e4ff34-7b79-4e55-a3a8-29a06758e550-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.448509 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.448488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cx75\" (UniqueName: \"kubernetes.io/projected/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kube-api-access-7cx75\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.939405 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.939367    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e4ff34-7b79-4e55-a3a8-29a06758e550-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:50.941900 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:50.941867    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e4ff34-7b79-4e55-a3a8-29a06758e550-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:51.115774 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:51.115746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:58:51.404924 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:51.404884    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"]
Sep 16 18:58:51.407625 ip-10-0-129-50 kubenswrapper[2582]: W0916 18:58:51.407592    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e4ff34_7b79_4e55_a3a8_29a06758e550.slice/crio-a80e491ee34580d8931bc2b839f26f466df941be3174994f10517539bbe2cbd8 WatchSource:0}: Error finding container a80e491ee34580d8931bc2b839f26f466df941be3174994f10517539bbe2cbd8: Status 404 returned error can't find the container with id a80e491ee34580d8931bc2b839f26f466df941be3174994f10517539bbe2cbd8
Sep 16 18:58:52.292381 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:52.292330    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 18:58:52.292570 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:52.292456    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:58:52.292670 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:52.292646    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:52.292778 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:52.292764    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:58:52.360445 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:52.360415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" event={"ID":"d1e4ff34-7b79-4e55-a3a8-29a06758e550","Type":"ContainerStarted","Data":"affeec9d4557dca25d779e40ac5c1fea15eceb7b2d5bbda6ceba86c3f6ada718"}
Sep 16 18:58:52.360567 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:52.360467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" event={"ID":"d1e4ff34-7b79-4e55-a3a8-29a06758e550","Type":"ContainerStarted","Data":"a80e491ee34580d8931bc2b839f26f466df941be3174994f10517539bbe2cbd8"}
Sep 16 18:58:52.891517 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:52.891478    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 16 18:58:55.375060 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:55.374986    2582 generic.go:356] "Generic (PLEG): container finished" podID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerID="affeec9d4557dca25d779e40ac5c1fea15eceb7b2d5bbda6ceba86c3f6ada718" exitCode=0
Sep 16 18:58:55.375480 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:55.375057    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" event={"ID":"d1e4ff34-7b79-4e55-a3a8-29a06758e550","Type":"ContainerDied","Data":"affeec9d4557dca25d779e40ac5c1fea15eceb7b2d5bbda6ceba86c3f6ada718"}
Sep 16 18:58:55.376369 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:55.376351    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:58:57.891637 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:58:57.891359    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 16 18:59:00.420057 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.419814    2582 generic.go:356] "Generic (PLEG): container finished" podID="48145d0a-85c8-47de-b218-ab3bbafead09" containerID="679db500ad5e8e6c9f41e1356a5d246475c68b991c7cb182acb3bcd131c2e79c" exitCode=137
Sep 16 18:59:00.420057 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.419928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" event={"ID":"48145d0a-85c8-47de-b218-ab3bbafead09","Type":"ContainerDied","Data":"679db500ad5e8e6c9f41e1356a5d246475c68b991c7cb182acb3bcd131c2e79c"}
Sep 16 18:59:00.655804 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.655771    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:59:00.725942 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.725898    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48145d0a-85c8-47de-b218-ab3bbafead09-proxy-tls\") pod \"48145d0a-85c8-47de-b218-ab3bbafead09\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") "
Sep 16 18:59:00.726112 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.725989    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfshj\" (UniqueName: \"kubernetes.io/projected/48145d0a-85c8-47de-b218-ab3bbafead09-kube-api-access-vfshj\") pod \"48145d0a-85c8-47de-b218-ab3bbafead09\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") "
Sep 16 18:59:00.726182 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.726113    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48145d0a-85c8-47de-b218-ab3bbafead09-cabundle-cert\") pod \"48145d0a-85c8-47de-b218-ab3bbafead09\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") "
Sep 16 18:59:00.726182 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.726150    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48145d0a-85c8-47de-b218-ab3bbafead09-kserve-provision-location\") pod \"48145d0a-85c8-47de-b218-ab3bbafead09\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") "
Sep 16 18:59:00.726306 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.726245    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48145d0a-85c8-47de-b218-ab3bbafead09-kube-rbac-proxy-sar-config\") pod \"48145d0a-85c8-47de-b218-ab3bbafead09\" (UID: \"48145d0a-85c8-47de-b218-ab3bbafead09\") "
Sep 16 18:59:00.726588 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.726545    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48145d0a-85c8-47de-b218-ab3bbafead09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48145d0a-85c8-47de-b218-ab3bbafead09" (UID: "48145d0a-85c8-47de-b218-ab3bbafead09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:00.726730 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.726590    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48145d0a-85c8-47de-b218-ab3bbafead09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "48145d0a-85c8-47de-b218-ab3bbafead09" (UID: "48145d0a-85c8-47de-b218-ab3bbafead09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:00.726730 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.726627    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48145d0a-85c8-47de-b218-ab3bbafead09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "48145d0a-85c8-47de-b218-ab3bbafead09" (UID: "48145d0a-85c8-47de-b218-ab3bbafead09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:00.728798 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.728763    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48145d0a-85c8-47de-b218-ab3bbafead09-kube-api-access-vfshj" (OuterVolumeSpecName: "kube-api-access-vfshj") pod "48145d0a-85c8-47de-b218-ab3bbafead09" (UID: "48145d0a-85c8-47de-b218-ab3bbafead09"). InnerVolumeSpecName "kube-api-access-vfshj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:00.728932 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.728808    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48145d0a-85c8-47de-b218-ab3bbafead09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48145d0a-85c8-47de-b218-ab3bbafead09" (UID: "48145d0a-85c8-47de-b218-ab3bbafead09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:00.827119 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.827046    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48145d0a-85c8-47de-b218-ab3bbafead09-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:59:00.827119 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.827079    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48145d0a-85c8-47de-b218-ab3bbafead09-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:59:00.827119 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.827094    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48145d0a-85c8-47de-b218-ab3bbafead09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:59:00.827119 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.827107    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48145d0a-85c8-47de-b218-ab3bbafead09-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:59:00.827119 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:00.827120    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfshj\" (UniqueName: \"kubernetes.io/projected/48145d0a-85c8-47de-b218-ab3bbafead09-kube-api-access-vfshj\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 18:59:01.429936 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:01.429888    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"
Sep 16 18:59:01.430357 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:01.429948    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7" event={"ID":"48145d0a-85c8-47de-b218-ab3bbafead09","Type":"ContainerDied","Data":"35b7ce2b6152bfcd89fdb0f7be7decdd5d2a439cfd746e6de2947285fcf6fffe"}
Sep 16 18:59:01.430357 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:01.430033    2582 scope.go:117] "RemoveContainer" containerID="679db500ad5e8e6c9f41e1356a5d246475c68b991c7cb182acb3bcd131c2e79c"
Sep 16 18:59:01.462158 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:01.462120    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"]
Sep 16 18:59:01.474674 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:01.474629    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7"]
Sep 16 18:59:03.299615 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:03.299576    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" path="/var/lib/kubelet/pods/48145d0a-85c8-47de-b218-ab3bbafead09/volumes"
Sep 16 18:59:06.329856 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:06.329700    2582 scope.go:117] "RemoveContainer" containerID="d9b9806050c889463fba6d7b672dd24bbab83a7a41d351372a82947c9ef669a7"
Sep 16 18:59:06.426312 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:06.426277    2582 scope.go:117] "RemoveContainer" containerID="0c6394d3700e2f06782eb81bf2b49a589b43635943013dd5e25b013be17d7dfc"
Sep 16 18:59:06.436317 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:06.436298    2582 scope.go:117] "RemoveContainer" containerID="06c3204e51aaf2c1bc7f75d8a353bb8c67730a1d3ce97d9dc6bdd8ff3a44fb64"
Sep 16 18:59:17.527116 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:17.527078    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" event={"ID":"d1e4ff34-7b79-4e55-a3a8-29a06758e550","Type":"ContainerStarted","Data":"fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01"}
Sep 16 18:59:17.527501 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:17.527157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" event={"ID":"d1e4ff34-7b79-4e55-a3a8-29a06758e550","Type":"ContainerStarted","Data":"3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2"}
Sep 16 18:59:17.527501 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:17.527360    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:59:17.527501 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:17.527415    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:59:23.537141 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:23.537115    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 18:59:23.569674 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:23.569631    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" podStartSLOduration=8.751367892 podStartE2EDuration="33.569618756s" totalImagesPullingTime="21.231003932s" totalInitContainerRuntime="3.587246932s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:17.563930812 +0000 UTC m=+1152.751306592" watchObservedRunningTime="2026-09-16 18:59:23.569618756 +0000 UTC m=+1158.756994543"
Sep 16 18:59:53.538137 ip-10-0-129-50 kubenswrapper[2582]: I0916 18:59:53.538097    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:00:03.538111 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:03.538067    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:00:05.283679 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:05.283648    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:00:05.286266 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:05.286244    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:00:13.538500 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:13.538459    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:00:23.538543 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:23.538506    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:00:33.622057 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:33.622023    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 19:00:40.397498 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:40.397468    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"]
Sep 16 19:00:40.398026 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:40.397687    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kserve-container" containerID="cri-o://3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2" gracePeriod=30
Sep 16 19:00:40.398026 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:40.397734    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kube-rbac-proxy" containerID="cri-o://fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01" gracePeriod=30
Sep 16 19:00:40.906150 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:40.906119    2582 generic.go:356] "Generic (PLEG): container finished" podID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerID="fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01" exitCode=2
Sep 16 19:00:40.906279 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:40.906157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" event={"ID":"d1e4ff34-7b79-4e55-a3a8-29a06758e550","Type":"ContainerDied","Data":"fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01"}
Sep 16 19:00:43.532587 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:43.532540    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 16 19:00:43.538044 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:43.538019    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:00:44.235619 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.235589    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 19:00:44.409210 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.409138    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kube-rbac-proxy-sar-config\") pod \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") "
Sep 16 19:00:44.409358 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.409224    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kserve-provision-location\") pod \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") "
Sep 16 19:00:44.409358 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.409278    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e4ff34-7b79-4e55-a3a8-29a06758e550-cabundle-cert\") pod \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") "
Sep 16 19:00:44.409358 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.409324    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cx75\" (UniqueName: \"kubernetes.io/projected/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kube-api-access-7cx75\") pod \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") "
Sep 16 19:00:44.409519 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.409359    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e4ff34-7b79-4e55-a3a8-29a06758e550-proxy-tls\") pod \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\" (UID: \"d1e4ff34-7b79-4e55-a3a8-29a06758e550\") "
Sep 16 19:00:44.409577 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.409554    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1e4ff34-7b79-4e55-a3a8-29a06758e550" (UID: "d1e4ff34-7b79-4e55-a3a8-29a06758e550"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:44.409623 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.409487    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1e4ff34-7b79-4e55-a3a8-29a06758e550" (UID: "d1e4ff34-7b79-4e55-a3a8-29a06758e550"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:44.409670 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.409623    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:00:44.409670 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.409626    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e4ff34-7b79-4e55-a3a8-29a06758e550-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d1e4ff34-7b79-4e55-a3a8-29a06758e550" (UID: "d1e4ff34-7b79-4e55-a3a8-29a06758e550"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:44.411629 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.411602    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kube-api-access-7cx75" (OuterVolumeSpecName: "kube-api-access-7cx75") pod "d1e4ff34-7b79-4e55-a3a8-29a06758e550" (UID: "d1e4ff34-7b79-4e55-a3a8-29a06758e550"). InnerVolumeSpecName "kube-api-access-7cx75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:44.411629 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.411606    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e4ff34-7b79-4e55-a3a8-29a06758e550-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1e4ff34-7b79-4e55-a3a8-29a06758e550" (UID: "d1e4ff34-7b79-4e55-a3a8-29a06758e550"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:44.510732 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.510704    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:00:44.510732 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.510728    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e4ff34-7b79-4e55-a3a8-29a06758e550-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:00:44.510872 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.510741    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cx75\" (UniqueName: \"kubernetes.io/projected/d1e4ff34-7b79-4e55-a3a8-29a06758e550-kube-api-access-7cx75\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:00:44.510872 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.510754    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e4ff34-7b79-4e55-a3a8-29a06758e550-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:00:44.931571 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.931537    2582 generic.go:356] "Generic (PLEG): container finished" podID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerID="3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2" exitCode=0
Sep 16 19:00:44.931984 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.931610    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"
Sep 16 19:00:44.931984 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.931642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" event={"ID":"d1e4ff34-7b79-4e55-a3a8-29a06758e550","Type":"ContainerDied","Data":"3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2"}
Sep 16 19:00:44.931984 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.931734    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79" event={"ID":"d1e4ff34-7b79-4e55-a3a8-29a06758e550","Type":"ContainerDied","Data":"a80e491ee34580d8931bc2b839f26f466df941be3174994f10517539bbe2cbd8"}
Sep 16 19:00:44.931984 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.931763    2582 scope.go:117] "RemoveContainer" containerID="fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01"
Sep 16 19:00:44.943647 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.942056    2582 scope.go:117] "RemoveContainer" containerID="3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2"
Sep 16 19:00:44.951970 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.951952    2582 scope.go:117] "RemoveContainer" containerID="affeec9d4557dca25d779e40ac5c1fea15eceb7b2d5bbda6ceba86c3f6ada718"
Sep 16 19:00:44.960411 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.960392    2582 scope.go:117] "RemoveContainer" containerID="fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01"
Sep 16 19:00:44.960755 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:00:44.960735    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01\": container with ID starting with fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01 not found: ID does not exist" containerID="fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01"
Sep 16 19:00:44.960816 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.960762    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01"} err="failed to get container status \"fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01\": rpc error: code = NotFound desc = could not find container \"fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01\": container with ID starting with fcbd81e67ed5e80802bd198080f84b7a82bd16356c14a68c16a4d5d35f2b0c01 not found: ID does not exist"
Sep 16 19:00:44.960816 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.960779    2582 scope.go:117] "RemoveContainer" containerID="3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2"
Sep 16 19:00:44.961085 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:00:44.961064    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2\": container with ID starting with 3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2 not found: ID does not exist" containerID="3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2"
Sep 16 19:00:44.961169 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.961094    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2"} err="failed to get container status \"3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2\": rpc error: code = NotFound desc = could not find container \"3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2\": container with ID starting with 3487bb130968c740d9c7e38fcf9b0b8857b3e77ae5d792427041a4f9c16d2cf2 not found: ID does not exist"
Sep 16 19:00:44.961169 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.961114    2582 scope.go:117] "RemoveContainer" containerID="affeec9d4557dca25d779e40ac5c1fea15eceb7b2d5bbda6ceba86c3f6ada718"
Sep 16 19:00:44.961383 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:00:44.961367    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"affeec9d4557dca25d779e40ac5c1fea15eceb7b2d5bbda6ceba86c3f6ada718\": container with ID starting with affeec9d4557dca25d779e40ac5c1fea15eceb7b2d5bbda6ceba86c3f6ada718 not found: ID does not exist" containerID="affeec9d4557dca25d779e40ac5c1fea15eceb7b2d5bbda6ceba86c3f6ada718"
Sep 16 19:00:44.961422 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.961388    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affeec9d4557dca25d779e40ac5c1fea15eceb7b2d5bbda6ceba86c3f6ada718"} err="failed to get container status \"affeec9d4557dca25d779e40ac5c1fea15eceb7b2d5bbda6ceba86c3f6ada718\": rpc error: code = NotFound desc = could not find container \"affeec9d4557dca25d779e40ac5c1fea15eceb7b2d5bbda6ceba86c3f6ada718\": container with ID starting with affeec9d4557dca25d779e40ac5c1fea15eceb7b2d5bbda6ceba86c3f6ada718 not found: ID does not exist"
Sep 16 19:00:44.961600 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.961581    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"]
Sep 16 19:00:44.969829 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:44.969805    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79"]
Sep 16 19:00:45.296029 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.295966    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" path="/var/lib/kubelet/pods/d1e4ff34-7b79-4e55-a3a8-29a06758e550/volumes"
Sep 16 19:00:45.762380 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762351    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"]
Sep 16 19:00:45.762741 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762726    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent"
Sep 16 19:00:45.762741 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762742    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent"
Sep 16 19:00:45.762833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762761    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kserve-container"
Sep 16 19:00:45.762833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762768    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kserve-container"
Sep 16 19:00:45.762833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762775    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="storage-initializer"
Sep 16 19:00:45.762833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762781    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="storage-initializer"
Sep 16 19:00:45.762833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762788    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kube-rbac-proxy"
Sep 16 19:00:45.762833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762793    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kube-rbac-proxy"
Sep 16 19:00:45.762833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762801    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container"
Sep 16 19:00:45.762833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762807    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container"
Sep 16 19:00:45.762833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762813    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="storage-initializer"
Sep 16 19:00:45.762833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762818    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="storage-initializer"
Sep 16 19:00:45.762833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762826    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kube-rbac-proxy"
Sep 16 19:00:45.762833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762831    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kube-rbac-proxy"
Sep 16 19:00:45.763232 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762884    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kserve-container"
Sep 16 19:00:45.763232 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762892    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kserve-container"
Sep 16 19:00:45.763232 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762899    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="agent"
Sep 16 19:00:45.763232 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762926    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e4ff34-7b79-4e55-a3a8-29a06758e550" containerName="kube-rbac-proxy"
Sep 16 19:00:45.763232 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.762932    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="48145d0a-85c8-47de-b218-ab3bbafead09" containerName="kube-rbac-proxy"
Sep 16 19:00:45.766337 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.766318    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:45.772499 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.772472    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:00:45.772648 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.772521    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:00:45.773221 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.773199    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 16 19:00:45.773303 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.773205    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:00:45.780044 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.780008    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:00:45.780153 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.780006    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:00:45.780352 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.780338    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:00:45.810642 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.810609    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"]
Sep 16 19:00:45.920367 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.920341    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79878dcf-7e0b-45db-81b1-186e61cc63d0-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:45.920475 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.920380    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79878dcf-7e0b-45db-81b1-186e61cc63d0-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:45.920475 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.920448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79878dcf-7e0b-45db-81b1-186e61cc63d0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:45.920567 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.920509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwbw9\" (UniqueName: \"kubernetes.io/projected/79878dcf-7e0b-45db-81b1-186e61cc63d0-kube-api-access-hwbw9\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:45.920567 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:45.920532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79878dcf-7e0b-45db-81b1-186e61cc63d0-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:46.021598 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.021535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwbw9\" (UniqueName: \"kubernetes.io/projected/79878dcf-7e0b-45db-81b1-186e61cc63d0-kube-api-access-hwbw9\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:46.021598 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.021573    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79878dcf-7e0b-45db-81b1-186e61cc63d0-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:46.022059 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.021617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79878dcf-7e0b-45db-81b1-186e61cc63d0-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:46.022059 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.021658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79878dcf-7e0b-45db-81b1-186e61cc63d0-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:46.022059 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.021706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79878dcf-7e0b-45db-81b1-186e61cc63d0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:46.022186 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.022085    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79878dcf-7e0b-45db-81b1-186e61cc63d0-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:46.022442 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.022404    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79878dcf-7e0b-45db-81b1-186e61cc63d0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:46.022442 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.022429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79878dcf-7e0b-45db-81b1-186e61cc63d0-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:46.024489 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.024469    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79878dcf-7e0b-45db-81b1-186e61cc63d0-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:46.044573 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.044547    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwbw9\" (UniqueName: \"kubernetes.io/projected/79878dcf-7e0b-45db-81b1-186e61cc63d0-kube-api-access-hwbw9\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:46.076801 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.076775    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:46.370173 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:00:46.370144    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79878dcf_7e0b_45db_81b1_186e61cc63d0.slice/crio-11a444820ab4ff172f75ef8c27f90b51c75a1481b1f1beb2c199bfa63d64d15f WatchSource:0}: Error finding container 11a444820ab4ff172f75ef8c27f90b51c75a1481b1f1beb2c199bfa63d64d15f: Status 404 returned error can't find the container with id 11a444820ab4ff172f75ef8c27f90b51c75a1481b1f1beb2c199bfa63d64d15f
Sep 16 19:00:46.380412 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.380392    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"]
Sep 16 19:00:46.944885 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.944852    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" event={"ID":"79878dcf-7e0b-45db-81b1-186e61cc63d0","Type":"ContainerStarted","Data":"b5e03f7886d95714cc306bde5d22a1fd742d4d6749805652d173acffb8742ad6"}
Sep 16 19:00:46.945049 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:46.944934    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" event={"ID":"79878dcf-7e0b-45db-81b1-186e61cc63d0","Type":"ContainerStarted","Data":"11a444820ab4ff172f75ef8c27f90b51c75a1481b1f1beb2c199bfa63d64d15f"}
Sep 16 19:00:50.966767 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:50.966727    2582 generic.go:356] "Generic (PLEG): container finished" podID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerID="b5e03f7886d95714cc306bde5d22a1fd742d4d6749805652d173acffb8742ad6" exitCode=0
Sep 16 19:00:50.967367 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:50.966810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" event={"ID":"79878dcf-7e0b-45db-81b1-186e61cc63d0","Type":"ContainerDied","Data":"b5e03f7886d95714cc306bde5d22a1fd742d4d6749805652d173acffb8742ad6"}
Sep 16 19:00:51.973467 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:51.973430    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" event={"ID":"79878dcf-7e0b-45db-81b1-186e61cc63d0","Type":"ContainerStarted","Data":"4c1582366267b59b1c3b4a8e24ea7dd9156fd6f8d48e916179014e1e5dddf0ff"}
Sep 16 19:00:51.973831 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:51.973496    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" event={"ID":"79878dcf-7e0b-45db-81b1-186e61cc63d0","Type":"ContainerStarted","Data":"6d520e4e17108b69193f9427c1f06ab7a721388390603cc4972bfec9ec409639"}
Sep 16 19:00:51.973831 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:51.973686    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:52.006806 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:52.006755    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" podStartSLOduration=3.260780306 podStartE2EDuration="7.006740912s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.745960606s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:52.005872593 +0000 UTC m=+1247.193248384" watchObservedRunningTime="2026-09-16 19:00:52.006740912 +0000 UTC m=+1247.194116699"
Sep 16 19:00:52.977887 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:52.977860    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:00:58.989090 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:00:58.989016    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:01:28.990515 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:01:28.990466    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 16 19:01:38.990338 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:01:38.990299    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 16 19:01:48.989754 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:01:48.989713    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 16 19:01:59.021802 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:01:59.021772    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:02:05.551855 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:05.551818    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"]
Sep 16 19:02:05.552283 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:05.552057    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kserve-container" containerID="cri-o://6d520e4e17108b69193f9427c1f06ab7a721388390603cc4972bfec9ec409639" gracePeriod=30
Sep 16 19:02:05.552283 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:05.552083    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kube-rbac-proxy" containerID="cri-o://4c1582366267b59b1c3b4a8e24ea7dd9156fd6f8d48e916179014e1e5dddf0ff" gracePeriod=30
Sep 16 19:02:06.319048 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:06.319015    2582 generic.go:356] "Generic (PLEG): container finished" podID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerID="4c1582366267b59b1c3b4a8e24ea7dd9156fd6f8d48e916179014e1e5dddf0ff" exitCode=2
Sep 16 19:02:06.319207 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:06.319073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" event={"ID":"79878dcf-7e0b-45db-81b1-186e61cc63d0","Type":"ContainerDied","Data":"4c1582366267b59b1c3b4a8e24ea7dd9156fd6f8d48e916179014e1e5dddf0ff"}
Sep 16 19:02:08.984370 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:08.984323    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 16 19:02:08.989843 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:08.989810    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 16 19:02:09.261726 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:02:09.261686    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79878dcf_7e0b_45db_81b1_186e61cc63d0.slice/crio-conmon-6d520e4e17108b69193f9427c1f06ab7a721388390603cc4972bfec9ec409639.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79878dcf_7e0b_45db_81b1_186e61cc63d0.slice/crio-6d520e4e17108b69193f9427c1f06ab7a721388390603cc4972bfec9ec409639.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:02:09.337011 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.336980    2582 generic.go:356] "Generic (PLEG): container finished" podID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerID="6d520e4e17108b69193f9427c1f06ab7a721388390603cc4972bfec9ec409639" exitCode=0
Sep 16 19:02:09.337141 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.337051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" event={"ID":"79878dcf-7e0b-45db-81b1-186e61cc63d0","Type":"ContainerDied","Data":"6d520e4e17108b69193f9427c1f06ab7a721388390603cc4972bfec9ec409639"}
Sep 16 19:02:09.397313 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.397292    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:02:09.527434 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.527370    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79878dcf-7e0b-45db-81b1-186e61cc63d0-kube-rbac-proxy-sar-config\") pod \"79878dcf-7e0b-45db-81b1-186e61cc63d0\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") "
Sep 16 19:02:09.527434 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.527424    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79878dcf-7e0b-45db-81b1-186e61cc63d0-proxy-tls\") pod \"79878dcf-7e0b-45db-81b1-186e61cc63d0\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") "
Sep 16 19:02:09.527631 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.527458    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79878dcf-7e0b-45db-81b1-186e61cc63d0-cabundle-cert\") pod \"79878dcf-7e0b-45db-81b1-186e61cc63d0\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") "
Sep 16 19:02:09.527631 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.527490    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwbw9\" (UniqueName: \"kubernetes.io/projected/79878dcf-7e0b-45db-81b1-186e61cc63d0-kube-api-access-hwbw9\") pod \"79878dcf-7e0b-45db-81b1-186e61cc63d0\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") "
Sep 16 19:02:09.527631 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.527519    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79878dcf-7e0b-45db-81b1-186e61cc63d0-kserve-provision-location\") pod \"79878dcf-7e0b-45db-81b1-186e61cc63d0\" (UID: \"79878dcf-7e0b-45db-81b1-186e61cc63d0\") "
Sep 16 19:02:09.527898 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.527776    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79878dcf-7e0b-45db-81b1-186e61cc63d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79878dcf-7e0b-45db-81b1-186e61cc63d0" (UID: "79878dcf-7e0b-45db-81b1-186e61cc63d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:09.527898 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.527874    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79878dcf-7e0b-45db-81b1-186e61cc63d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79878dcf-7e0b-45db-81b1-186e61cc63d0" (UID: "79878dcf-7e0b-45db-81b1-186e61cc63d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:09.528067 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.527892    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79878dcf-7e0b-45db-81b1-186e61cc63d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79878dcf-7e0b-45db-81b1-186e61cc63d0" (UID: "79878dcf-7e0b-45db-81b1-186e61cc63d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:09.529688 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.529665    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79878dcf-7e0b-45db-81b1-186e61cc63d0-kube-api-access-hwbw9" (OuterVolumeSpecName: "kube-api-access-hwbw9") pod "79878dcf-7e0b-45db-81b1-186e61cc63d0" (UID: "79878dcf-7e0b-45db-81b1-186e61cc63d0"). InnerVolumeSpecName "kube-api-access-hwbw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:09.529773 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.529740    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79878dcf-7e0b-45db-81b1-186e61cc63d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79878dcf-7e0b-45db-81b1-186e61cc63d0" (UID: "79878dcf-7e0b-45db-81b1-186e61cc63d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:09.628852 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.628832    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwbw9\" (UniqueName: \"kubernetes.io/projected/79878dcf-7e0b-45db-81b1-186e61cc63d0-kube-api-access-hwbw9\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:02:09.628852 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.628852    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79878dcf-7e0b-45db-81b1-186e61cc63d0-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:02:09.628997 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.628862    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79878dcf-7e0b-45db-81b1-186e61cc63d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:02:09.628997 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.628870    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79878dcf-7e0b-45db-81b1-186e61cc63d0-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:02:09.628997 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:09.628880    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79878dcf-7e0b-45db-81b1-186e61cc63d0-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:02:10.345565 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.345538    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"
Sep 16 19:02:10.346200 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.345562    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4" event={"ID":"79878dcf-7e0b-45db-81b1-186e61cc63d0","Type":"ContainerDied","Data":"11a444820ab4ff172f75ef8c27f90b51c75a1481b1f1beb2c199bfa63d64d15f"}
Sep 16 19:02:10.346200 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.345631    2582 scope.go:117] "RemoveContainer" containerID="4c1582366267b59b1c3b4a8e24ea7dd9156fd6f8d48e916179014e1e5dddf0ff"
Sep 16 19:02:10.356181 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.356162    2582 scope.go:117] "RemoveContainer" containerID="6d520e4e17108b69193f9427c1f06ab7a721388390603cc4972bfec9ec409639"
Sep 16 19:02:10.364435 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.364410    2582 scope.go:117] "RemoveContainer" containerID="b5e03f7886d95714cc306bde5d22a1fd742d4d6749805652d173acffb8742ad6"
Sep 16 19:02:10.371372 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.371335    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"]
Sep 16 19:02:10.374327 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.374268    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4"]
Sep 16 19:02:10.795940 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.795894    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"]
Sep 16 19:02:10.796271 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.796259    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kube-rbac-proxy"
Sep 16 19:02:10.796318 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.796273    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kube-rbac-proxy"
Sep 16 19:02:10.796318 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.796284    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="storage-initializer"
Sep 16 19:02:10.796318 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.796290    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="storage-initializer"
Sep 16 19:02:10.796318 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.796298    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kserve-container"
Sep 16 19:02:10.796318 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.796303    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kserve-container"
Sep 16 19:02:10.796512 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.796356    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kserve-container"
Sep 16 19:02:10.796512 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.796368    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" containerName="kube-rbac-proxy"
Sep 16 19:02:10.800760 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.800743    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:10.804597 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.804580    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:02:10.804867 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.804839    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:02:10.804867 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.804854    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 16 19:02:10.804867 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.804840    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:02:10.805119 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.804877    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:02:10.805178 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.805152    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:02:10.808325 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.808306    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:02:10.814428 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.814407    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"]
Sep 16 19:02:10.937604 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.937580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:10.937705 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.937627    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fca01ad-ef32-4026-9193-06ee2a90c7d8-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:10.937705 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.937649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwcdt\" (UniqueName: \"kubernetes.io/projected/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kube-api-access-nwcdt\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:10.937705 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.937672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fca01ad-ef32-4026-9193-06ee2a90c7d8-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:10.937813 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:10.937719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:11.038618 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.038585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:11.038712 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.038633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:11.038712 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.038676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fca01ad-ef32-4026-9193-06ee2a90c7d8-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:11.038712 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.038695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwcdt\" (UniqueName: \"kubernetes.io/projected/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kube-api-access-nwcdt\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:11.038817 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.038711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fca01ad-ef32-4026-9193-06ee2a90c7d8-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:11.038865 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:02:11.038824    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 16 19:02:11.038970 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:02:11.038955    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fca01ad-ef32-4026-9193-06ee2a90c7d8-proxy-tls podName:1fca01ad-ef32-4026-9193-06ee2a90c7d8 nodeName:}" failed. No retries permitted until 2026-09-16 19:02:11.538898824 +0000 UTC m=+1326.726274590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1fca01ad-ef32-4026-9193-06ee2a90c7d8-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" (UID: "1fca01ad-ef32-4026-9193-06ee2a90c7d8") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 16 19:02:11.039058 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.038973    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:11.039370 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.039344    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fca01ad-ef32-4026-9193-06ee2a90c7d8-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:11.039418 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.039370    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:11.052932 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.052863    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwcdt\" (UniqueName: \"kubernetes.io/projected/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kube-api-access-nwcdt\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:11.296945 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.296899    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79878dcf-7e0b-45db-81b1-186e61cc63d0" path="/var/lib/kubelet/pods/79878dcf-7e0b-45db-81b1-186e61cc63d0/volumes"
Sep 16 19:02:11.542915 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.542884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fca01ad-ef32-4026-9193-06ee2a90c7d8-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:11.545618 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.545595    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fca01ad-ef32-4026-9193-06ee2a90c7d8-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:11.711225 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:11.711196    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:12.004226 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:12.004203    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"]
Sep 16 19:02:12.006769 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:02:12.006729    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fca01ad_ef32_4026_9193_06ee2a90c7d8.slice/crio-63d373a2b9ccf07b85cdf2489332907618490fc7342f473b660ffe8beb3e68f3 WatchSource:0}: Error finding container 63d373a2b9ccf07b85cdf2489332907618490fc7342f473b660ffe8beb3e68f3: Status 404 returned error can't find the container with id 63d373a2b9ccf07b85cdf2489332907618490fc7342f473b660ffe8beb3e68f3
Sep 16 19:02:12.359420 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:12.359337    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" event={"ID":"1fca01ad-ef32-4026-9193-06ee2a90c7d8","Type":"ContainerStarted","Data":"fb66eb77aa26cfdd04d7304ec11b5215bfb85c4005fa78e565a5f60582fc715d"}
Sep 16 19:02:12.359420 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:12.359407    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" event={"ID":"1fca01ad-ef32-4026-9193-06ee2a90c7d8","Type":"ContainerStarted","Data":"63d373a2b9ccf07b85cdf2489332907618490fc7342f473b660ffe8beb3e68f3"}
Sep 16 19:02:16.381943 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:16.381842    2582 generic.go:356] "Generic (PLEG): container finished" podID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerID="fb66eb77aa26cfdd04d7304ec11b5215bfb85c4005fa78e565a5f60582fc715d" exitCode=0
Sep 16 19:02:16.381943 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:16.381929    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" event={"ID":"1fca01ad-ef32-4026-9193-06ee2a90c7d8","Type":"ContainerDied","Data":"fb66eb77aa26cfdd04d7304ec11b5215bfb85c4005fa78e565a5f60582fc715d"}
Sep 16 19:02:17.389740 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:17.389710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" event={"ID":"1fca01ad-ef32-4026-9193-06ee2a90c7d8","Type":"ContainerStarted","Data":"be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a"}
Sep 16 19:02:17.390123 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:17.389764    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" event={"ID":"1fca01ad-ef32-4026-9193-06ee2a90c7d8","Type":"ContainerStarted","Data":"56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1"}
Sep 16 19:02:17.390123 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:17.389945    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:17.424938 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:17.424871    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" podStartSLOduration=3.504357305 podStartE2EDuration="7.424853626s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.920496321s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:17.423270754 +0000 UTC m=+1332.610646555" watchObservedRunningTime="2026-09-16 19:02:17.424853626 +0000 UTC m=+1332.612229412"
Sep 16 19:02:18.394565 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:18.394533    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:24.404759 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:24.404729    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:02:54.406164 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:02:54.406126    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 16 19:03:04.405186 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:04.405152    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 16 19:03:14.405801 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:14.405761    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 16 19:03:24.405353 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:24.405317    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 16 19:03:30.322428 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:30.322398    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:03:34.867181 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:34.867150    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"]
Sep 16 19:03:34.867553 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:34.867393    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kserve-container" containerID="cri-o://56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1" gracePeriod=30
Sep 16 19:03:34.867553 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:34.867444    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kube-rbac-proxy" containerID="cri-o://be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a" gracePeriod=30
Sep 16 19:03:35.741621 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:35.741589    2582 generic.go:356] "Generic (PLEG): container finished" podID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerID="be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a" exitCode=2
Sep 16 19:03:35.741793 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:35.741653    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" event={"ID":"1fca01ad-ef32-4026-9193-06ee2a90c7d8","Type":"ContainerDied","Data":"be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a"}
Sep 16 19:03:39.399399 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:39.399350    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 16 19:03:40.291936 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.291882    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 16 19:03:40.508612 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.508591    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:03:40.610400 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.610346    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fca01ad-ef32-4026-9193-06ee2a90c7d8-cabundle-cert\") pod \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") "
Sep 16 19:03:40.610400 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.610391    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kube-rbac-proxy-sar-config\") pod \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") "
Sep 16 19:03:40.610596 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.610423    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fca01ad-ef32-4026-9193-06ee2a90c7d8-proxy-tls\") pod \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") "
Sep 16 19:03:40.610596 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.610441    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwcdt\" (UniqueName: \"kubernetes.io/projected/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kube-api-access-nwcdt\") pod \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") "
Sep 16 19:03:40.610596 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.610485    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kserve-provision-location\") pod \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\" (UID: \"1fca01ad-ef32-4026-9193-06ee2a90c7d8\") "
Sep 16 19:03:40.610882 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.610856    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fca01ad-ef32-4026-9193-06ee2a90c7d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1fca01ad-ef32-4026-9193-06ee2a90c7d8" (UID: "1fca01ad-ef32-4026-9193-06ee2a90c7d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:40.610882 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.610873    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1fca01ad-ef32-4026-9193-06ee2a90c7d8" (UID: "1fca01ad-ef32-4026-9193-06ee2a90c7d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:40.611049 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.610892    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1fca01ad-ef32-4026-9193-06ee2a90c7d8" (UID: "1fca01ad-ef32-4026-9193-06ee2a90c7d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:40.612610 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.612590    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fca01ad-ef32-4026-9193-06ee2a90c7d8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1fca01ad-ef32-4026-9193-06ee2a90c7d8" (UID: "1fca01ad-ef32-4026-9193-06ee2a90c7d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:40.612744 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.612723    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kube-api-access-nwcdt" (OuterVolumeSpecName: "kube-api-access-nwcdt") pod "1fca01ad-ef32-4026-9193-06ee2a90c7d8" (UID: "1fca01ad-ef32-4026-9193-06ee2a90c7d8"). InnerVolumeSpecName "kube-api-access-nwcdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:40.711192 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.711164    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:03:40.711192 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.711188    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fca01ad-ef32-4026-9193-06ee2a90c7d8-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:03:40.711322 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.711201    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:03:40.711322 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.711216    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fca01ad-ef32-4026-9193-06ee2a90c7d8-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:03:40.711322 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.711229    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwcdt\" (UniqueName: \"kubernetes.io/projected/1fca01ad-ef32-4026-9193-06ee2a90c7d8-kube-api-access-nwcdt\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:03:40.765276 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.765250    2582 generic.go:356] "Generic (PLEG): container finished" podID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerID="56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1" exitCode=0
Sep 16 19:03:40.765370 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.765303    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"
Sep 16 19:03:40.765414 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.765303    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" event={"ID":"1fca01ad-ef32-4026-9193-06ee2a90c7d8","Type":"ContainerDied","Data":"56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1"}
Sep 16 19:03:40.765447 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.765433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4" event={"ID":"1fca01ad-ef32-4026-9193-06ee2a90c7d8","Type":"ContainerDied","Data":"63d373a2b9ccf07b85cdf2489332907618490fc7342f473b660ffe8beb3e68f3"}
Sep 16 19:03:40.765484 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.765465    2582 scope.go:117] "RemoveContainer" containerID="be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a"
Sep 16 19:03:40.777780 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.777763    2582 scope.go:117] "RemoveContainer" containerID="56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1"
Sep 16 19:03:40.785277 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.785260    2582 scope.go:117] "RemoveContainer" containerID="fb66eb77aa26cfdd04d7304ec11b5215bfb85c4005fa78e565a5f60582fc715d"
Sep 16 19:03:40.792487 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.792471    2582 scope.go:117] "RemoveContainer" containerID="be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a"
Sep 16 19:03:40.792759 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:03:40.792742    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a\": container with ID starting with be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a not found: ID does not exist" containerID="be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a"
Sep 16 19:03:40.792823 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.792767    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a"} err="failed to get container status \"be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a\": rpc error: code = NotFound desc = could not find container \"be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a\": container with ID starting with be879a5207595ad64ee9c2b0f00fdb9913a35a19db75358a7571789f6b324a7a not found: ID does not exist"
Sep 16 19:03:40.792823 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.792783    2582 scope.go:117] "RemoveContainer" containerID="56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1"
Sep 16 19:03:40.793032 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:03:40.793018    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1\": container with ID starting with 56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1 not found: ID does not exist" containerID="56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1"
Sep 16 19:03:40.793083 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.793035    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1"} err="failed to get container status \"56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1\": rpc error: code = NotFound desc = could not find container \"56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1\": container with ID starting with 56476c981684596763cce53efd6bc4cd75d73fc3b87ac98307217107700399d1 not found: ID does not exist"
Sep 16 19:03:40.793083 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.793046    2582 scope.go:117] "RemoveContainer" containerID="fb66eb77aa26cfdd04d7304ec11b5215bfb85c4005fa78e565a5f60582fc715d"
Sep 16 19:03:40.793276 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:03:40.793261    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb66eb77aa26cfdd04d7304ec11b5215bfb85c4005fa78e565a5f60582fc715d\": container with ID starting with fb66eb77aa26cfdd04d7304ec11b5215bfb85c4005fa78e565a5f60582fc715d not found: ID does not exist" containerID="fb66eb77aa26cfdd04d7304ec11b5215bfb85c4005fa78e565a5f60582fc715d"
Sep 16 19:03:40.793321 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.793279    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb66eb77aa26cfdd04d7304ec11b5215bfb85c4005fa78e565a5f60582fc715d"} err="failed to get container status \"fb66eb77aa26cfdd04d7304ec11b5215bfb85c4005fa78e565a5f60582fc715d\": rpc error: code = NotFound desc = could not find container \"fb66eb77aa26cfdd04d7304ec11b5215bfb85c4005fa78e565a5f60582fc715d\": container with ID starting with fb66eb77aa26cfdd04d7304ec11b5215bfb85c4005fa78e565a5f60582fc715d not found: ID does not exist"
Sep 16 19:03:40.798704 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.798685    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"]
Sep 16 19:03:40.803555 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:40.803537    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4"]
Sep 16 19:03:41.297393 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:41.297367    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" path="/var/lib/kubelet/pods/1fca01ad-ef32-4026-9193-06ee2a90c7d8/volumes"
Sep 16 19:03:57.610144 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.610056    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-1438bdf6"]
Sep 16 19:03:57.610618 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.610421    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kube-rbac-proxy"
Sep 16 19:03:57.610618 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.610436    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kube-rbac-proxy"
Sep 16 19:03:57.610618 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.610462    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="storage-initializer"
Sep 16 19:03:57.610618 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.610467    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="storage-initializer"
Sep 16 19:03:57.610618 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.610473    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kserve-container"
Sep 16 19:03:57.610618 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.610478    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kserve-container"
Sep 16 19:03:57.610618 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.610532    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kserve-container"
Sep 16 19:03:57.610618 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.610540    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fca01ad-ef32-4026-9193-06ee2a90c7d8" containerName="kube-rbac-proxy"
Sep 16 19:03:57.615230 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.615211    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-1438bdf6"
Sep 16 19:03:57.618563 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.618537    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 19:03:57.619362 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.619342    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2q5x2\""
Sep 16 19:03:57.619472 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.619342    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 19:03:57.624620 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.624594    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-1438bdf6"]
Sep 16 19:03:57.739922 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.739878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rkx\" (UniqueName: \"kubernetes.io/projected/8992e8a7-0ada-4ae0-bb90-0258e80d32b2-kube-api-access-64rkx\") pod \"curl-metrics-1438bdf6\" (UID: \"8992e8a7-0ada-4ae0-bb90-0258e80d32b2\") " pod="kserve/curl-metrics-1438bdf6"
Sep 16 19:03:57.841118 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.841092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rkx\" (UniqueName: \"kubernetes.io/projected/8992e8a7-0ada-4ae0-bb90-0258e80d32b2-kube-api-access-64rkx\") pod \"curl-metrics-1438bdf6\" (UID: \"8992e8a7-0ada-4ae0-bb90-0258e80d32b2\") " pod="kserve/curl-metrics-1438bdf6"
Sep 16 19:03:57.854126 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.854106    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rkx\" (UniqueName: \"kubernetes.io/projected/8992e8a7-0ada-4ae0-bb90-0258e80d32b2-kube-api-access-64rkx\") pod \"curl-metrics-1438bdf6\" (UID: \"8992e8a7-0ada-4ae0-bb90-0258e80d32b2\") " pod="kserve/curl-metrics-1438bdf6"
Sep 16 19:03:57.939030 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:57.939007    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-1438bdf6"
Sep 16 19:03:58.214232 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:58.214210    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-1438bdf6"]
Sep 16 19:03:58.216730 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:03:58.216703    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8992e8a7_0ada_4ae0_bb90_0258e80d32b2.slice/crio-fbf804d1389ed8f1544bc69e6c8df1f35aba2917bab0182c3af7859b0e847210 WatchSource:0}: Error finding container fbf804d1389ed8f1544bc69e6c8df1f35aba2917bab0182c3af7859b0e847210: Status 404 returned error can't find the container with id fbf804d1389ed8f1544bc69e6c8df1f35aba2917bab0182c3af7859b0e847210
Sep 16 19:03:58.218598 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:58.218575    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:03:58.844399 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:58.844371    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-1438bdf6" event={"ID":"8992e8a7-0ada-4ae0-bb90-0258e80d32b2","Type":"ContainerStarted","Data":"fbf804d1389ed8f1544bc69e6c8df1f35aba2917bab0182c3af7859b0e847210"}
Sep 16 19:03:59.851555 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:59.851520    2582 generic.go:356] "Generic (PLEG): container finished" podID="8992e8a7-0ada-4ae0-bb90-0258e80d32b2" containerID="2a55979ca7b05456dafac30c10d0f2c2e1e3d6a48a420ef7d27ba774695dde14" exitCode=0
Sep 16 19:03:59.851996 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:03:59.851610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-1438bdf6" event={"ID":"8992e8a7-0ada-4ae0-bb90-0258e80d32b2","Type":"ContainerDied","Data":"2a55979ca7b05456dafac30c10d0f2c2e1e3d6a48a420ef7d27ba774695dde14"}
Sep 16 19:04:00.993612 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:00.993587    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-1438bdf6"
Sep 16 19:04:01.061605 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:01.061573    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-1438bdf6_8992e8a7-0ada-4ae0-bb90-0258e80d32b2/curl-metrics-1438bdf6/0.log"
Sep 16 19:04:01.064953 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:01.064935    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64rkx\" (UniqueName: \"kubernetes.io/projected/8992e8a7-0ada-4ae0-bb90-0258e80d32b2-kube-api-access-64rkx\") pod \"8992e8a7-0ada-4ae0-bb90-0258e80d32b2\" (UID: \"8992e8a7-0ada-4ae0-bb90-0258e80d32b2\") "
Sep 16 19:04:01.066959 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:01.066940    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8992e8a7-0ada-4ae0-bb90-0258e80d32b2-kube-api-access-64rkx" (OuterVolumeSpecName: "kube-api-access-64rkx") pod "8992e8a7-0ada-4ae0-bb90-0258e80d32b2" (UID: "8992e8a7-0ada-4ae0-bb90-0258e80d32b2"). InnerVolumeSpecName "kube-api-access-64rkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:01.096303 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:01.096277    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-1438bdf6"]
Sep 16 19:04:01.097739 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:01.097719    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-1438bdf6"]
Sep 16 19:04:01.165655 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:01.165602    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64rkx\" (UniqueName: \"kubernetes.io/projected/8992e8a7-0ada-4ae0-bb90-0258e80d32b2-kube-api-access-64rkx\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:04:01.295890 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:01.295864    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8992e8a7-0ada-4ae0-bb90-0258e80d32b2" path="/var/lib/kubelet/pods/8992e8a7-0ada-4ae0-bb90-0258e80d32b2/volumes"
Sep 16 19:04:01.863871 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:01.863848    2582 scope.go:117] "RemoveContainer" containerID="2a55979ca7b05456dafac30c10d0f2c2e1e3d6a48a420ef7d27ba774695dde14"
Sep 16 19:04:01.863871 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:01.863860    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-1438bdf6"
Sep 16 19:04:06.516265 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.516225    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-669c4320"]
Sep 16 19:04:06.516648 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.516590    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8992e8a7-0ada-4ae0-bb90-0258e80d32b2" containerName="curl-metrics-1438bdf6"
Sep 16 19:04:06.516648 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.516603    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8992e8a7-0ada-4ae0-bb90-0258e80d32b2" containerName="curl-metrics-1438bdf6"
Sep 16 19:04:06.516729 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.516690    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8992e8a7-0ada-4ae0-bb90-0258e80d32b2" containerName="curl-metrics-1438bdf6"
Sep 16 19:04:06.520009 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.519990    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-669c4320"
Sep 16 19:04:06.525185 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.525160    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 19:04:06.526019 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.525999    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2q5x2\""
Sep 16 19:04:06.526113 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.526068    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 19:04:06.534496 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.534467    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-669c4320"]
Sep 16 19:04:06.607228 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.607203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fgcp\" (UniqueName: \"kubernetes.io/projected/0b2e3945-1013-4366-8fdc-61dc730af296-kube-api-access-8fgcp\") pod \"curl-metrics-669c4320\" (UID: \"0b2e3945-1013-4366-8fdc-61dc730af296\") " pod="kserve/curl-metrics-669c4320"
Sep 16 19:04:06.708336 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.708312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fgcp\" (UniqueName: \"kubernetes.io/projected/0b2e3945-1013-4366-8fdc-61dc730af296-kube-api-access-8fgcp\") pod \"curl-metrics-669c4320\" (UID: \"0b2e3945-1013-4366-8fdc-61dc730af296\") " pod="kserve/curl-metrics-669c4320"
Sep 16 19:04:06.720189 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.720159    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fgcp\" (UniqueName: \"kubernetes.io/projected/0b2e3945-1013-4366-8fdc-61dc730af296-kube-api-access-8fgcp\") pod \"curl-metrics-669c4320\" (UID: \"0b2e3945-1013-4366-8fdc-61dc730af296\") " pod="kserve/curl-metrics-669c4320"
Sep 16 19:04:06.841412 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:06.841347    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-669c4320"
Sep 16 19:04:07.126280 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:07.126215    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-669c4320"]
Sep 16 19:04:07.129413 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:04:07.129385    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b2e3945_1013_4366_8fdc_61dc730af296.slice/crio-16a3ff122ac5d81c04861e72c6030f220735756ef50fff562a64393132461381 WatchSource:0}: Error finding container 16a3ff122ac5d81c04861e72c6030f220735756ef50fff562a64393132461381: Status 404 returned error can't find the container with id 16a3ff122ac5d81c04861e72c6030f220735756ef50fff562a64393132461381
Sep 16 19:04:07.893984 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:07.893948    2582 generic.go:356] "Generic (PLEG): container finished" podID="0b2e3945-1013-4366-8fdc-61dc730af296" containerID="e6eda8ccb164abe561c2bf87b7a045da212a63d9f138a7726b1158107e1c241c" exitCode=0
Sep 16 19:04:07.894322 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:07.894025    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-669c4320" event={"ID":"0b2e3945-1013-4366-8fdc-61dc730af296","Type":"ContainerDied","Data":"e6eda8ccb164abe561c2bf87b7a045da212a63d9f138a7726b1158107e1c241c"}
Sep 16 19:04:07.894322 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:07.894069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-669c4320" event={"ID":"0b2e3945-1013-4366-8fdc-61dc730af296","Type":"ContainerStarted","Data":"16a3ff122ac5d81c04861e72c6030f220735756ef50fff562a64393132461381"}
Sep 16 19:04:09.034429 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:09.034404    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-669c4320"
Sep 16 19:04:09.063232 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:09.063212    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-669c4320_0b2e3945-1013-4366-8fdc-61dc730af296/curl-metrics-669c4320/0.log"
Sep 16 19:04:09.087288 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:09.087267    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-669c4320"]
Sep 16 19:04:09.090694 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:09.090673    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-669c4320"]
Sep 16 19:04:09.130109 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:09.130089    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fgcp\" (UniqueName: \"kubernetes.io/projected/0b2e3945-1013-4366-8fdc-61dc730af296-kube-api-access-8fgcp\") pod \"0b2e3945-1013-4366-8fdc-61dc730af296\" (UID: \"0b2e3945-1013-4366-8fdc-61dc730af296\") "
Sep 16 19:04:09.132384 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:09.132359    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2e3945-1013-4366-8fdc-61dc730af296-kube-api-access-8fgcp" (OuterVolumeSpecName: "kube-api-access-8fgcp") pod "0b2e3945-1013-4366-8fdc-61dc730af296" (UID: "0b2e3945-1013-4366-8fdc-61dc730af296"). InnerVolumeSpecName "kube-api-access-8fgcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:09.231451 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:09.231432    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fgcp\" (UniqueName: \"kubernetes.io/projected/0b2e3945-1013-4366-8fdc-61dc730af296-kube-api-access-8fgcp\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:04:09.296375 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:09.296352    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b2e3945-1013-4366-8fdc-61dc730af296" path="/var/lib/kubelet/pods/0b2e3945-1013-4366-8fdc-61dc730af296/volumes"
Sep 16 19:04:09.912797 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:09.912771    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-669c4320"
Sep 16 19:04:09.912975 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:09.912817    2582 scope.go:117] "RemoveContainer" containerID="e6eda8ccb164abe561c2bf87b7a045da212a63d9f138a7726b1158107e1c241c"
Sep 16 19:04:14.537066 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.537029    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"]
Sep 16 19:04:14.537432 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.537391    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b2e3945-1013-4366-8fdc-61dc730af296" containerName="curl-metrics-669c4320"
Sep 16 19:04:14.537432 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.537401    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b2e3945-1013-4366-8fdc-61dc730af296" containerName="curl-metrics-669c4320"
Sep 16 19:04:14.537503 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.537455    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b2e3945-1013-4366-8fdc-61dc730af296" containerName="curl-metrics-669c4320"
Sep 16 19:04:14.541157 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.541139    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.547610 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.544460    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 16 19:04:14.548648 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.548621    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:04:14.548865 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.548844    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:04:14.549069 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.549048    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:04:14.549139 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.549100    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 16 19:04:14.549620 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.549603    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:04:14.549832 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.549817    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:04:14.556686 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.556661    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"]
Sep 16 19:04:14.569674 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.569653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/817abde1-00ac-4296-a65d-7373a0e6135b-proxy-tls\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.569788 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.569709    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/817abde1-00ac-4296-a65d-7373a0e6135b-kserve-provision-location\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.569848 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.569833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/817abde1-00ac-4296-a65d-7373a0e6135b-cabundle-cert\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.569997 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.569960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/817abde1-00ac-4296-a65d-7373a0e6135b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.570328 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.569994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvc4k\" (UniqueName: \"kubernetes.io/projected/817abde1-00ac-4296-a65d-7373a0e6135b-kube-api-access-vvc4k\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.671040 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.671019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/817abde1-00ac-4296-a65d-7373a0e6135b-proxy-tls\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.671149 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.671057    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/817abde1-00ac-4296-a65d-7373a0e6135b-kserve-provision-location\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.671149 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.671096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/817abde1-00ac-4296-a65d-7373a0e6135b-cabundle-cert\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.671149 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.671130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/817abde1-00ac-4296-a65d-7373a0e6135b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.671149 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.671146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvc4k\" (UniqueName: \"kubernetes.io/projected/817abde1-00ac-4296-a65d-7373a0e6135b-kube-api-access-vvc4k\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.671531 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.671507    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/817abde1-00ac-4296-a65d-7373a0e6135b-kserve-provision-location\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.671806 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.671784    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/817abde1-00ac-4296-a65d-7373a0e6135b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.671874 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.671814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/817abde1-00ac-4296-a65d-7373a0e6135b-cabundle-cert\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.673636 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.673619    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/817abde1-00ac-4296-a65d-7373a0e6135b-proxy-tls\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.681453 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.681428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvc4k\" (UniqueName: \"kubernetes.io/projected/817abde1-00ac-4296-a65d-7373a0e6135b-kube-api-access-vvc4k\") pod \"isvc-sklearn-predictor-78d7749847-5xhdr\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:14.856381 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:14.856332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:15.135776 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:15.135755    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"]
Sep 16 19:04:15.137844 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:04:15.137815    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817abde1_00ac_4296_a65d_7373a0e6135b.slice/crio-75320be4d0d0d14370d33c8588d01b93d8b410b819ef1f62fce5755fad7dbae2 WatchSource:0}: Error finding container 75320be4d0d0d14370d33c8588d01b93d8b410b819ef1f62fce5755fad7dbae2: Status 404 returned error can't find the container with id 75320be4d0d0d14370d33c8588d01b93d8b410b819ef1f62fce5755fad7dbae2
Sep 16 19:04:15.943187 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:15.943148    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" event={"ID":"817abde1-00ac-4296-a65d-7373a0e6135b","Type":"ContainerStarted","Data":"fa303805e3465bcf28f12c10c8e684406cca93df6194da3621de2ca1ba06847b"}
Sep 16 19:04:15.943658 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:15.943214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" event={"ID":"817abde1-00ac-4296-a65d-7373a0e6135b","Type":"ContainerStarted","Data":"75320be4d0d0d14370d33c8588d01b93d8b410b819ef1f62fce5755fad7dbae2"}
Sep 16 19:04:19.965100 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:19.965066    2582 generic.go:356] "Generic (PLEG): container finished" podID="817abde1-00ac-4296-a65d-7373a0e6135b" containerID="fa303805e3465bcf28f12c10c8e684406cca93df6194da3621de2ca1ba06847b" exitCode=0
Sep 16 19:04:19.965592 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:19.965160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" event={"ID":"817abde1-00ac-4296-a65d-7373a0e6135b","Type":"ContainerDied","Data":"fa303805e3465bcf28f12c10c8e684406cca93df6194da3621de2ca1ba06847b"}
Sep 16 19:04:20.972548 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:20.972515    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" event={"ID":"817abde1-00ac-4296-a65d-7373a0e6135b","Type":"ContainerStarted","Data":"78184ce479adbdf720b433a1769f481b8cefffbd8b29ce0aa5315667313c260b"}
Sep 16 19:04:20.972939 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:20.972574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" event={"ID":"817abde1-00ac-4296-a65d-7373a0e6135b","Type":"ContainerStarted","Data":"bbee8cd1fbcc7833d6bc22d9bd41673ea6b009890d50941e91cc85c4e37e13c8"}
Sep 16 19:04:20.972939 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:20.972752    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:20.996816 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:20.996771    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podStartSLOduration=3.045474355 podStartE2EDuration="6.996758403s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.951284048s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:20.994682908 +0000 UTC m=+1456.182058721" watchObservedRunningTime="2026-09-16 19:04:20.996758403 +0000 UTC m=+1456.184134189"
Sep 16 19:04:21.977208 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:21.977175    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:21.978420 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:21.978393    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:04:22.982681 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:22.982641    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:04:27.987107 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:27.987081    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:04:27.987593 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:27.987565    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:04:37.987994 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:37.987953    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:04:47.987978 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:47.987941    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:04:57.988293 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:04:57.988251    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:05:05.321369 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:05.321344    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:05:05.325381 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:05.325358    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:05:07.987991 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:07.987951    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:05:17.987623 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:17.987583    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:05:27.988469 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:27.988385    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:05:37.988568 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:37.988536    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:05:44.730230 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:44.730194    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"]
Sep 16 19:05:44.732810 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:44.730439    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container" containerID="cri-o://bbee8cd1fbcc7833d6bc22d9bd41673ea6b009890d50941e91cc85c4e37e13c8" gracePeriod=30
Sep 16 19:05:44.732810 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:44.730495    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kube-rbac-proxy" containerID="cri-o://78184ce479adbdf720b433a1769f481b8cefffbd8b29ce0aa5315667313c260b" gracePeriod=30
Sep 16 19:05:45.354101 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:45.354066    2582 generic.go:356] "Generic (PLEG): container finished" podID="817abde1-00ac-4296-a65d-7373a0e6135b" containerID="78184ce479adbdf720b433a1769f481b8cefffbd8b29ce0aa5315667313c260b" exitCode=2
Sep 16 19:05:45.354305 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:45.354128    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" event={"ID":"817abde1-00ac-4296-a65d-7373a0e6135b","Type":"ContainerDied","Data":"78184ce479adbdf720b433a1769f481b8cefffbd8b29ce0aa5315667313c260b"}
Sep 16 19:05:47.983283 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:47.983238    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 16 19:05:47.987546 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:47.987514    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:05:48.372845 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.372803    2582 generic.go:356] "Generic (PLEG): container finished" podID="817abde1-00ac-4296-a65d-7373a0e6135b" containerID="bbee8cd1fbcc7833d6bc22d9bd41673ea6b009890d50941e91cc85c4e37e13c8" exitCode=0
Sep 16 19:05:48.373008 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.372858    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" event={"ID":"817abde1-00ac-4296-a65d-7373a0e6135b","Type":"ContainerDied","Data":"bbee8cd1fbcc7833d6bc22d9bd41673ea6b009890d50941e91cc85c4e37e13c8"}
Sep 16 19:05:48.476524 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.476500    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:05:48.562913 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.562888    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/817abde1-00ac-4296-a65d-7373a0e6135b-kube-rbac-proxy-sar-config\") pod \"817abde1-00ac-4296-a65d-7373a0e6135b\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") "
Sep 16 19:05:48.563051 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.562930    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/817abde1-00ac-4296-a65d-7373a0e6135b-cabundle-cert\") pod \"817abde1-00ac-4296-a65d-7373a0e6135b\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") "
Sep 16 19:05:48.563051 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.562964    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvc4k\" (UniqueName: \"kubernetes.io/projected/817abde1-00ac-4296-a65d-7373a0e6135b-kube-api-access-vvc4k\") pod \"817abde1-00ac-4296-a65d-7373a0e6135b\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") "
Sep 16 19:05:48.563051 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.563006    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/817abde1-00ac-4296-a65d-7373a0e6135b-kserve-provision-location\") pod \"817abde1-00ac-4296-a65d-7373a0e6135b\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") "
Sep 16 19:05:48.563225 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.563052    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/817abde1-00ac-4296-a65d-7373a0e6135b-proxy-tls\") pod \"817abde1-00ac-4296-a65d-7373a0e6135b\" (UID: \"817abde1-00ac-4296-a65d-7373a0e6135b\") "
Sep 16 19:05:48.563333 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.563309    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817abde1-00ac-4296-a65d-7373a0e6135b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "817abde1-00ac-4296-a65d-7373a0e6135b" (UID: "817abde1-00ac-4296-a65d-7373a0e6135b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:48.563404 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.563320    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817abde1-00ac-4296-a65d-7373a0e6135b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "817abde1-00ac-4296-a65d-7373a0e6135b" (UID: "817abde1-00ac-4296-a65d-7373a0e6135b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:48.563404 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.563337    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817abde1-00ac-4296-a65d-7373a0e6135b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "817abde1-00ac-4296-a65d-7373a0e6135b" (UID: "817abde1-00ac-4296-a65d-7373a0e6135b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:48.565274 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.565248    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/817abde1-00ac-4296-a65d-7373a0e6135b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "817abde1-00ac-4296-a65d-7373a0e6135b" (UID: "817abde1-00ac-4296-a65d-7373a0e6135b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:48.565274 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.565255    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817abde1-00ac-4296-a65d-7373a0e6135b-kube-api-access-vvc4k" (OuterVolumeSpecName: "kube-api-access-vvc4k") pod "817abde1-00ac-4296-a65d-7373a0e6135b" (UID: "817abde1-00ac-4296-a65d-7373a0e6135b"). InnerVolumeSpecName "kube-api-access-vvc4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:48.663889 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.663863    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/817abde1-00ac-4296-a65d-7373a0e6135b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:05:48.663889 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.663885    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/817abde1-00ac-4296-a65d-7373a0e6135b-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:05:48.664035 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.663895    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvc4k\" (UniqueName: \"kubernetes.io/projected/817abde1-00ac-4296-a65d-7373a0e6135b-kube-api-access-vvc4k\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:05:48.664035 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.663926    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/817abde1-00ac-4296-a65d-7373a0e6135b-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:05:48.664035 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:48.663936    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/817abde1-00ac-4296-a65d-7373a0e6135b-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:05:49.379890 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.379865    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"
Sep 16 19:05:49.380272 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.379856    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr" event={"ID":"817abde1-00ac-4296-a65d-7373a0e6135b","Type":"ContainerDied","Data":"75320be4d0d0d14370d33c8588d01b93d8b410b819ef1f62fce5755fad7dbae2"}
Sep 16 19:05:49.380272 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.379995    2582 scope.go:117] "RemoveContainer" containerID="78184ce479adbdf720b433a1769f481b8cefffbd8b29ce0aa5315667313c260b"
Sep 16 19:05:49.391598 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.391582    2582 scope.go:117] "RemoveContainer" containerID="bbee8cd1fbcc7833d6bc22d9bd41673ea6b009890d50941e91cc85c4e37e13c8"
Sep 16 19:05:49.400658 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.400053    2582 scope.go:117] "RemoveContainer" containerID="fa303805e3465bcf28f12c10c8e684406cca93df6194da3621de2ca1ba06847b"
Sep 16 19:05:49.402863 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.402844    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"]
Sep 16 19:05:49.408231 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.408207    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr"]
Sep 16 19:05:49.901876 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.901848    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"]
Sep 16 19:05:49.902178 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.902167    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="storage-initializer"
Sep 16 19:05:49.902226 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.902180    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="storage-initializer"
Sep 16 19:05:49.902226 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.902189    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container"
Sep 16 19:05:49.902226 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.902194    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container"
Sep 16 19:05:49.902226 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.902206    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kube-rbac-proxy"
Sep 16 19:05:49.902226 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.902212    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kube-rbac-proxy"
Sep 16 19:05:49.902380 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.902278    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kube-rbac-proxy"
Sep 16 19:05:49.902380 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.902285    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" containerName="kserve-container"
Sep 16 19:05:49.906770 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.906754    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:49.909505 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.909486    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:05:49.909773 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.909744    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:05:49.909833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.909755    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 16 19:05:49.910675 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.910658    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:05:49.910739 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.910723    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:05:49.910826 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.910813    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:05:49.911051 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.911035    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:05:49.927722 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.927702    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"]
Sep 16 19:05:49.974601 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.974572    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6755acaf-ff23-49f6-8d9e-40b2d2c90658-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:49.974716 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.974645    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8c6\" (UniqueName: \"kubernetes.io/projected/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kube-api-access-xp8c6\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:49.974716 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.974674    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:49.974716 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.974702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:49.974866 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:49.974765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6755acaf-ff23-49f6-8d9e-40b2d2c90658-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:50.075622 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:50.075604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6755acaf-ff23-49f6-8d9e-40b2d2c90658-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:50.075748 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:50.075635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6755acaf-ff23-49f6-8d9e-40b2d2c90658-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:50.075748 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:50.075676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8c6\" (UniqueName: \"kubernetes.io/projected/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kube-api-access-xp8c6\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:50.075748 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:50.075693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:50.075748 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:50.075711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:50.076055 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:50.076036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:50.076346 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:50.076326    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:50.076422 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:50.076377    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6755acaf-ff23-49f6-8d9e-40b2d2c90658-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:50.078331 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:50.078312    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6755acaf-ff23-49f6-8d9e-40b2d2c90658-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:50.087953 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:50.087932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8c6\" (UniqueName: \"kubernetes.io/projected/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kube-api-access-xp8c6\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:50.216855 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:50.216835    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:05:50.500068 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:50.500047    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"]
Sep 16 19:05:50.502415 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:05:50.502387    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6755acaf_ff23_49f6_8d9e_40b2d2c90658.slice/crio-fd7d0a1ffbd923ff8b2601375c93878e8efb28e76c60ec6db5a3923275183c23 WatchSource:0}: Error finding container fd7d0a1ffbd923ff8b2601375c93878e8efb28e76c60ec6db5a3923275183c23: Status 404 returned error can't find the container with id fd7d0a1ffbd923ff8b2601375c93878e8efb28e76c60ec6db5a3923275183c23
Sep 16 19:05:51.296256 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:51.296226    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="817abde1-00ac-4296-a65d-7373a0e6135b" path="/var/lib/kubelet/pods/817abde1-00ac-4296-a65d-7373a0e6135b/volumes"
Sep 16 19:05:51.398368 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:51.398333    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" event={"ID":"6755acaf-ff23-49f6-8d9e-40b2d2c90658","Type":"ContainerStarted","Data":"0c8b77874e7a2cf281da64841ed68e4d08e901f12c5ea3af1df34b0283920e99"}
Sep 16 19:05:51.398527 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:51.398407    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" event={"ID":"6755acaf-ff23-49f6-8d9e-40b2d2c90658","Type":"ContainerStarted","Data":"fd7d0a1ffbd923ff8b2601375c93878e8efb28e76c60ec6db5a3923275183c23"}
Sep 16 19:05:54.413487 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:54.413457    2582 generic.go:356] "Generic (PLEG): container finished" podID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerID="0c8b77874e7a2cf281da64841ed68e4d08e901f12c5ea3af1df34b0283920e99" exitCode=0
Sep 16 19:05:54.413800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:05:54.413525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" event={"ID":"6755acaf-ff23-49f6-8d9e-40b2d2c90658","Type":"ContainerDied","Data":"0c8b77874e7a2cf281da64841ed68e4d08e901f12c5ea3af1df34b0283920e99"}
Sep 16 19:08:06.102817 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:08:06.102792    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" event={"ID":"6755acaf-ff23-49f6-8d9e-40b2d2c90658","Type":"ContainerStarted","Data":"f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582"}
Sep 16 19:08:07.109005 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:08:07.108969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" event={"ID":"6755acaf-ff23-49f6-8d9e-40b2d2c90658","Type":"ContainerStarted","Data":"54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f"}
Sep 16 19:08:07.109389 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:08:07.109220    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:08:07.109389 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:08:07.109246    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:08:07.141229 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:08:07.141185    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" podStartSLOduration=2.747055965 podStartE2EDuration="2m18.141165125s" totalImagesPullingTime="2m11.565414935s" totalInitContainerRuntime="3.828694225s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:07.139297809 +0000 UTC m=+1682.326673609" watchObservedRunningTime="2026-09-16 19:08:07.141165125 +0000 UTC m=+1682.328540912"
Sep 16 19:08:13.118627 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:08:13.118594    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:08:43.122261 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:08:43.122179    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:08:53.122446 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:08:53.122410    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:09:00.018824 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:00.018789    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"]
Sep 16 19:09:00.019341 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:00.019036    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="kserve-container" containerID="cri-o://f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582" gracePeriod=30
Sep 16 19:09:00.019341 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:00.019129    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="kube-rbac-proxy" containerID="cri-o://54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f" gracePeriod=30
Sep 16 19:09:00.347192 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:00.347123    2582 generic.go:356] "Generic (PLEG): container finished" podID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerID="54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f" exitCode=2
Sep 16 19:09:00.347291 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:00.347194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" event={"ID":"6755acaf-ff23-49f6-8d9e-40b2d2c90658","Type":"ContainerDied","Data":"54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f"}
Sep 16 19:09:03.113582 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:03.113544    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 16 19:09:03.120094 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:03.120056    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 16 19:09:05.246687 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.246658    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"]
Sep 16 19:09:05.249292 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.249276    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.251740 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.251716    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:09:05.251874 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.251717    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 16 19:09:05.266573 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.266555    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"]
Sep 16 19:09:05.298958 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.297045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/770a19f0-6bde-4292-b822-acfc1f70cd34-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.298958 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.297114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/770a19f0-6bde-4292-b822-acfc1f70cd34-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.298958 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.297174    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770a19f0-6bde-4292-b822-acfc1f70cd34-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.298958 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.297238    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/770a19f0-6bde-4292-b822-acfc1f70cd34-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.298958 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.297269    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vcsw\" (UniqueName: \"kubernetes.io/projected/770a19f0-6bde-4292-b822-acfc1f70cd34-kube-api-access-6vcsw\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.398515 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.398490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/770a19f0-6bde-4292-b822-acfc1f70cd34-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.398667 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.398530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/770a19f0-6bde-4292-b822-acfc1f70cd34-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.398667 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.398566    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770a19f0-6bde-4292-b822-acfc1f70cd34-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.398796 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:09:05.398664    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 16 19:09:05.398796 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:09:05.398753    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/770a19f0-6bde-4292-b822-acfc1f70cd34-proxy-tls podName:770a19f0-6bde-4292-b822-acfc1f70cd34 nodeName:}" failed. No retries permitted until 2026-09-16 19:09:05.898721522 +0000 UTC m=+1741.086097305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/770a19f0-6bde-4292-b822-acfc1f70cd34-proxy-tls") pod "isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" (UID: "770a19f0-6bde-4292-b822-acfc1f70cd34") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 16 19:09:05.398968 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.398945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/770a19f0-6bde-4292-b822-acfc1f70cd34-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.399040 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.398985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vcsw\" (UniqueName: \"kubernetes.io/projected/770a19f0-6bde-4292-b822-acfc1f70cd34-kube-api-access-6vcsw\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.399040 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.398998    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770a19f0-6bde-4292-b822-acfc1f70cd34-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.399344 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.399327    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/770a19f0-6bde-4292-b822-acfc1f70cd34-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.399557 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.399540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/770a19f0-6bde-4292-b822-acfc1f70cd34-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.410805 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.410784    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vcsw\" (UniqueName: \"kubernetes.io/projected/770a19f0-6bde-4292-b822-acfc1f70cd34-kube-api-access-6vcsw\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.761098 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.761077    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:09:05.802734 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.802711    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6755acaf-ff23-49f6-8d9e-40b2d2c90658-proxy-tls\") pod \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") "
Sep 16 19:09:05.802851 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.802750    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp8c6\" (UniqueName: \"kubernetes.io/projected/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kube-api-access-xp8c6\") pod \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") "
Sep 16 19:09:05.802851 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.802784    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kserve-provision-location\") pod \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") "
Sep 16 19:09:05.802980 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.802929    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kube-rbac-proxy-sar-config\") pod \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") "
Sep 16 19:09:05.803041 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.803007    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6755acaf-ff23-49f6-8d9e-40b2d2c90658-cabundle-cert\") pod \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\" (UID: \"6755acaf-ff23-49f6-8d9e-40b2d2c90658\") "
Sep 16 19:09:05.803169 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.803144    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6755acaf-ff23-49f6-8d9e-40b2d2c90658" (UID: "6755acaf-ff23-49f6-8d9e-40b2d2c90658"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:05.803357 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.803332    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6755acaf-ff23-49f6-8d9e-40b2d2c90658" (UID: "6755acaf-ff23-49f6-8d9e-40b2d2c90658"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:05.803423 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.803342    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:09:05.803614 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.803591    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6755acaf-ff23-49f6-8d9e-40b2d2c90658-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6755acaf-ff23-49f6-8d9e-40b2d2c90658" (UID: "6755acaf-ff23-49f6-8d9e-40b2d2c90658"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:05.804941 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.804920    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6755acaf-ff23-49f6-8d9e-40b2d2c90658-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6755acaf-ff23-49f6-8d9e-40b2d2c90658" (UID: "6755acaf-ff23-49f6-8d9e-40b2d2c90658"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:05.805071 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.805052    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kube-api-access-xp8c6" (OuterVolumeSpecName: "kube-api-access-xp8c6") pod "6755acaf-ff23-49f6-8d9e-40b2d2c90658" (UID: "6755acaf-ff23-49f6-8d9e-40b2d2c90658"). InnerVolumeSpecName "kube-api-access-xp8c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:05.904446 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.904397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/770a19f0-6bde-4292-b822-acfc1f70cd34-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:05.904563 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.904465    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6755acaf-ff23-49f6-8d9e-40b2d2c90658-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:09:05.904563 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.904482    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp8c6\" (UniqueName: \"kubernetes.io/projected/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kube-api-access-xp8c6\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:09:05.904563 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.904496    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6755acaf-ff23-49f6-8d9e-40b2d2c90658-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:09:05.904563 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.904508    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6755acaf-ff23-49f6-8d9e-40b2d2c90658-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:09:05.906782 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:05.906761    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/770a19f0-6bde-4292-b822-acfc1f70cd34-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7f586df5dc-mdl94\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:06.159510 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.159453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:06.374028 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.373998    2582 generic.go:356] "Generic (PLEG): container finished" podID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerID="f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582" exitCode=0
Sep 16 19:09:06.374380 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.374051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" event={"ID":"6755acaf-ff23-49f6-8d9e-40b2d2c90658","Type":"ContainerDied","Data":"f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582"}
Sep 16 19:09:06.374380 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.374062    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"
Sep 16 19:09:06.374380 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.374095    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4" event={"ID":"6755acaf-ff23-49f6-8d9e-40b2d2c90658","Type":"ContainerDied","Data":"fd7d0a1ffbd923ff8b2601375c93878e8efb28e76c60ec6db5a3923275183c23"}
Sep 16 19:09:06.374380 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.374121    2582 scope.go:117] "RemoveContainer" containerID="54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f"
Sep 16 19:09:06.382434 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.382411    2582 scope.go:117] "RemoveContainer" containerID="f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582"
Sep 16 19:09:06.389873 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.389856    2582 scope.go:117] "RemoveContainer" containerID="0c8b77874e7a2cf281da64841ed68e4d08e901f12c5ea3af1df34b0283920e99"
Sep 16 19:09:06.397291 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.397274    2582 scope.go:117] "RemoveContainer" containerID="54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f"
Sep 16 19:09:06.397542 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:09:06.397523    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f\": container with ID starting with 54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f not found: ID does not exist" containerID="54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f"
Sep 16 19:09:06.397587 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.397551    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f"} err="failed to get container status \"54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f\": rpc error: code = NotFound desc = could not find container \"54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f\": container with ID starting with 54d2584fb36fc80f23cf65447937110cc9818c5cfb6a5d94753d1682e78bdd7f not found: ID does not exist"
Sep 16 19:09:06.397587 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.397568    2582 scope.go:117] "RemoveContainer" containerID="f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582"
Sep 16 19:09:06.397789 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:09:06.397769    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582\": container with ID starting with f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582 not found: ID does not exist" containerID="f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582"
Sep 16 19:09:06.397829 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.397788    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582"} err="failed to get container status \"f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582\": rpc error: code = NotFound desc = could not find container \"f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582\": container with ID starting with f67efc9f4129c74c06a21aeda4a306046feb7ee1a0049d5fe62490cf7a989582 not found: ID does not exist"
Sep 16 19:09:06.397829 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.397799    2582 scope.go:117] "RemoveContainer" containerID="0c8b77874e7a2cf281da64841ed68e4d08e901f12c5ea3af1df34b0283920e99"
Sep 16 19:09:06.398037 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:09:06.398019    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8b77874e7a2cf281da64841ed68e4d08e901f12c5ea3af1df34b0283920e99\": container with ID starting with 0c8b77874e7a2cf281da64841ed68e4d08e901f12c5ea3af1df34b0283920e99 not found: ID does not exist" containerID="0c8b77874e7a2cf281da64841ed68e4d08e901f12c5ea3af1df34b0283920e99"
Sep 16 19:09:06.398100 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.398042    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8b77874e7a2cf281da64841ed68e4d08e901f12c5ea3af1df34b0283920e99"} err="failed to get container status \"0c8b77874e7a2cf281da64841ed68e4d08e901f12c5ea3af1df34b0283920e99\": rpc error: code = NotFound desc = could not find container \"0c8b77874e7a2cf281da64841ed68e4d08e901f12c5ea3af1df34b0283920e99\": container with ID starting with 0c8b77874e7a2cf281da64841ed68e4d08e901f12c5ea3af1df34b0283920e99 not found: ID does not exist"
Sep 16 19:09:06.401243 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.401220    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"]
Sep 16 19:09:06.418977 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.418923    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4"]
Sep 16 19:09:06.439781 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:06.439764    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"]
Sep 16 19:09:06.442105 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:09:06.442085    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770a19f0_6bde_4292_b822_acfc1f70cd34.slice/crio-a8028d1f94651f737d8cce095f5667e89a5a595d81ab897a20c27ae97ff36f7d WatchSource:0}: Error finding container a8028d1f94651f737d8cce095f5667e89a5a595d81ab897a20c27ae97ff36f7d: Status 404 returned error can't find the container with id a8028d1f94651f737d8cce095f5667e89a5a595d81ab897a20c27ae97ff36f7d
Sep 16 19:09:07.296517 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:07.296486    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" path="/var/lib/kubelet/pods/6755acaf-ff23-49f6-8d9e-40b2d2c90658/volumes"
Sep 16 19:09:07.381492 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:07.381454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" event={"ID":"770a19f0-6bde-4292-b822-acfc1f70cd34","Type":"ContainerStarted","Data":"5b729840f5dd5caa75162008cb15bc0e919f4595c74bc89d4c2dd3b9c8092477"}
Sep 16 19:09:07.381945 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:07.381511    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" event={"ID":"770a19f0-6bde-4292-b822-acfc1f70cd34","Type":"ContainerStarted","Data":"a8028d1f94651f737d8cce095f5667e89a5a595d81ab897a20c27ae97ff36f7d"}
Sep 16 19:09:12.404041 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:12.404010    2582 generic.go:356] "Generic (PLEG): container finished" podID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerID="5b729840f5dd5caa75162008cb15bc0e919f4595c74bc89d4c2dd3b9c8092477" exitCode=0
Sep 16 19:09:12.404323 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:12.404071    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" event={"ID":"770a19f0-6bde-4292-b822-acfc1f70cd34","Type":"ContainerDied","Data":"5b729840f5dd5caa75162008cb15bc0e919f4595c74bc89d4c2dd3b9c8092477"}
Sep 16 19:09:13.410470 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:13.410426    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" event={"ID":"770a19f0-6bde-4292-b822-acfc1f70cd34","Type":"ContainerStarted","Data":"c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7"}
Sep 16 19:09:13.410921 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:13.410496    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" event={"ID":"770a19f0-6bde-4292-b822-acfc1f70cd34","Type":"ContainerStarted","Data":"7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b"}
Sep 16 19:09:13.410921 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:13.410692    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:13.435206 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:13.435156    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" podStartSLOduration=2.570684679 podStartE2EDuration="8.435141306s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.864456627s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:13.433647292 +0000 UTC m=+1748.621023075" watchObservedRunningTime="2026-09-16 19:09:13.435141306 +0000 UTC m=+1748.622517094"
Sep 16 19:09:14.422233 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:14.422191    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:14.423978 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:14.423939    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 16 19:09:15.426879 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:15.426837    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 16 19:09:20.432370 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:20.432337    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:20.432966 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:20.432817    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 16 19:09:30.434096 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:30.434064    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:37.242067 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:37.242029    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"]
Sep 16 19:09:37.242489 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:37.242284    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="kserve-container" containerID="cri-o://7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b" gracePeriod=30
Sep 16 19:09:37.242489 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:37.242340    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="kube-rbac-proxy" containerID="cri-o://c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7" gracePeriod=30
Sep 16 19:09:37.534736 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:37.534672    2582 generic.go:356] "Generic (PLEG): container finished" podID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerID="c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7" exitCode=2
Sep 16 19:09:37.534832 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:37.534749    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" event={"ID":"770a19f0-6bde-4292-b822-acfc1f70cd34","Type":"ContainerDied","Data":"c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7"}
Sep 16 19:09:38.372487 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.372465    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:38.465393 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.465368    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/770a19f0-6bde-4292-b822-acfc1f70cd34-kube-rbac-proxy-sar-config\") pod \"770a19f0-6bde-4292-b822-acfc1f70cd34\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") "
Sep 16 19:09:38.465513 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.465418    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vcsw\" (UniqueName: \"kubernetes.io/projected/770a19f0-6bde-4292-b822-acfc1f70cd34-kube-api-access-6vcsw\") pod \"770a19f0-6bde-4292-b822-acfc1f70cd34\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") "
Sep 16 19:09:38.465513 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.465446    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/770a19f0-6bde-4292-b822-acfc1f70cd34-proxy-tls\") pod \"770a19f0-6bde-4292-b822-acfc1f70cd34\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") "
Sep 16 19:09:38.465513 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.465488    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/770a19f0-6bde-4292-b822-acfc1f70cd34-cabundle-cert\") pod \"770a19f0-6bde-4292-b822-acfc1f70cd34\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") "
Sep 16 19:09:38.465681 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.465520    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770a19f0-6bde-4292-b822-acfc1f70cd34-kserve-provision-location\") pod \"770a19f0-6bde-4292-b822-acfc1f70cd34\" (UID: \"770a19f0-6bde-4292-b822-acfc1f70cd34\") "
Sep 16 19:09:38.465846 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.465816    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770a19f0-6bde-4292-b822-acfc1f70cd34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "770a19f0-6bde-4292-b822-acfc1f70cd34" (UID: "770a19f0-6bde-4292-b822-acfc1f70cd34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:38.465955 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.465844    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770a19f0-6bde-4292-b822-acfc1f70cd34-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "770a19f0-6bde-4292-b822-acfc1f70cd34" (UID: "770a19f0-6bde-4292-b822-acfc1f70cd34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:38.467692 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.467666    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770a19f0-6bde-4292-b822-acfc1f70cd34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "770a19f0-6bde-4292-b822-acfc1f70cd34" (UID: "770a19f0-6bde-4292-b822-acfc1f70cd34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:38.467787 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.467712    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770a19f0-6bde-4292-b822-acfc1f70cd34-kube-api-access-6vcsw" (OuterVolumeSpecName: "kube-api-access-6vcsw") pod "770a19f0-6bde-4292-b822-acfc1f70cd34" (UID: "770a19f0-6bde-4292-b822-acfc1f70cd34"). InnerVolumeSpecName "kube-api-access-6vcsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:38.492474 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.492414    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770a19f0-6bde-4292-b822-acfc1f70cd34-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "770a19f0-6bde-4292-b822-acfc1f70cd34" (UID: "770a19f0-6bde-4292-b822-acfc1f70cd34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:38.540564 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.540540    2582 generic.go:356] "Generic (PLEG): container finished" podID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerID="7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b" exitCode=0
Sep 16 19:09:38.540652 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.540582    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" event={"ID":"770a19f0-6bde-4292-b822-acfc1f70cd34","Type":"ContainerDied","Data":"7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b"}
Sep 16 19:09:38.540652 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.540592    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"
Sep 16 19:09:38.540652 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.540628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94" event={"ID":"770a19f0-6bde-4292-b822-acfc1f70cd34","Type":"ContainerDied","Data":"a8028d1f94651f737d8cce095f5667e89a5a595d81ab897a20c27ae97ff36f7d"}
Sep 16 19:09:38.540772 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.540654    2582 scope.go:117] "RemoveContainer" containerID="c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7"
Sep 16 19:09:38.549121 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.549105    2582 scope.go:117] "RemoveContainer" containerID="7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b"
Sep 16 19:09:38.556794 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.556775    2582 scope.go:117] "RemoveContainer" containerID="5b729840f5dd5caa75162008cb15bc0e919f4595c74bc89d4c2dd3b9c8092477"
Sep 16 19:09:38.564055 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.564038    2582 scope.go:117] "RemoveContainer" containerID="c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7"
Sep 16 19:09:38.564308 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:09:38.564290    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7\": container with ID starting with c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7 not found: ID does not exist" containerID="c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7"
Sep 16 19:09:38.564367 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.564315    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7"} err="failed to get container status \"c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7\": rpc error: code = NotFound desc = could not find container \"c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7\": container with ID starting with c79989da018ba891b8bdef0ea7a0c924490490af6bbd05ac55fa9f0268551dc7 not found: ID does not exist"
Sep 16 19:09:38.564367 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.564331    2582 scope.go:117] "RemoveContainer" containerID="7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b"
Sep 16 19:09:38.564583 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:09:38.564567    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b\": container with ID starting with 7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b not found: ID does not exist" containerID="7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b"
Sep 16 19:09:38.564634 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.564589    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b"} err="failed to get container status \"7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b\": rpc error: code = NotFound desc = could not find container \"7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b\": container with ID starting with 7d8e38d735beeca0222b235b3af64ef2fda6ba0dd4cc0f8064fbfb6ba3ef862b not found: ID does not exist"
Sep 16 19:09:38.564634 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.564603    2582 scope.go:117] "RemoveContainer" containerID="5b729840f5dd5caa75162008cb15bc0e919f4595c74bc89d4c2dd3b9c8092477"
Sep 16 19:09:38.564848 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:09:38.564833    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b729840f5dd5caa75162008cb15bc0e919f4595c74bc89d4c2dd3b9c8092477\": container with ID starting with 5b729840f5dd5caa75162008cb15bc0e919f4595c74bc89d4c2dd3b9c8092477 not found: ID does not exist" containerID="5b729840f5dd5caa75162008cb15bc0e919f4595c74bc89d4c2dd3b9c8092477"
Sep 16 19:09:38.564894 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.564852    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b729840f5dd5caa75162008cb15bc0e919f4595c74bc89d4c2dd3b9c8092477"} err="failed to get container status \"5b729840f5dd5caa75162008cb15bc0e919f4595c74bc89d4c2dd3b9c8092477\": rpc error: code = NotFound desc = could not find container \"5b729840f5dd5caa75162008cb15bc0e919f4595c74bc89d4c2dd3b9c8092477\": container with ID starting with 5b729840f5dd5caa75162008cb15bc0e919f4595c74bc89d4c2dd3b9c8092477 not found: ID does not exist"
Sep 16 19:09:38.566288 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.566267    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vcsw\" (UniqueName: \"kubernetes.io/projected/770a19f0-6bde-4292-b822-acfc1f70cd34-kube-api-access-6vcsw\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.566332 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.566297    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/770a19f0-6bde-4292-b822-acfc1f70cd34-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.566332 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.566312    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/770a19f0-6bde-4292-b822-acfc1f70cd34-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.566332 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.566324    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/770a19f0-6bde-4292-b822-acfc1f70cd34-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.566424 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.566332    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/770a19f0-6bde-4292-b822-acfc1f70cd34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.569845 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.569824    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"]
Sep 16 19:09:38.579306 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:38.579283    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94"]
Sep 16 19:09:39.296419 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:39.296395    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" path="/var/lib/kubelet/pods/770a19f0-6bde-4292-b822-acfc1f70cd34/volumes"
Sep 16 19:09:42.449797 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.449764    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"]
Sep 16 19:09:42.450307 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450174    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="kserve-container"
Sep 16 19:09:42.450307 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450190    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="kserve-container"
Sep 16 19:09:42.450307 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450207    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="kserve-container"
Sep 16 19:09:42.450307 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450217    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="kserve-container"
Sep 16 19:09:42.450307 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450229    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="kube-rbac-proxy"
Sep 16 19:09:42.450307 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450236    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="kube-rbac-proxy"
Sep 16 19:09:42.450307 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450251    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="kube-rbac-proxy"
Sep 16 19:09:42.450307 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450260    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="kube-rbac-proxy"
Sep 16 19:09:42.450307 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450272    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="storage-initializer"
Sep 16 19:09:42.450307 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450280    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="storage-initializer"
Sep 16 19:09:42.450307 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450287    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="storage-initializer"
Sep 16 19:09:42.450307 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450293    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="storage-initializer"
Sep 16 19:09:42.450729 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450359    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="kube-rbac-proxy"
Sep 16 19:09:42.450729 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450368    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="kube-rbac-proxy"
Sep 16 19:09:42.450729 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450376    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6755acaf-ff23-49f6-8d9e-40b2d2c90658" containerName="kserve-container"
Sep 16 19:09:42.450729 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.450381    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="770a19f0-6bde-4292-b822-acfc1f70cd34" containerName="kserve-container"
Sep 16 19:09:42.453895 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.453872    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.457143 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.457119    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 16 19:09:42.457688 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.457658    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:09:42.457865 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.457846    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:09:42.458010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.457994    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:09:42.458107 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.458081    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:09:42.458107 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.458088    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:09:42.458236 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.458212    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:09:42.468259 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.468234    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"]
Sep 16 19:09:42.599834 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.599811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxc76\" (UniqueName: \"kubernetes.io/projected/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kube-api-access-rxc76\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.600007 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.599881    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.600007 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.599953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.600007 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.599977    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.600007 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.600005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.700755 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.700693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.700755 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.700736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.700923 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.700765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.700923 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.700786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.700923 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.700805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxc76\" (UniqueName: \"kubernetes.io/projected/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kube-api-access-rxc76\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.701218 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.701200    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.701369 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.701352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.701427 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.701383    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.703395 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.703354    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.710928 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.710891    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxc76\" (UniqueName: \"kubernetes.io/projected/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kube-api-access-rxc76\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:42.766472 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:42.766449    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:43.046900 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:43.046878    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"]
Sep 16 19:09:43.048832 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:09:43.048805    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1de4a9_74b4_4a83_ad74_efca0fe02f8b.slice/crio-1ac4243caaa1ffe16e3f3e72d2d1d80bc81fbd5a8e63892b64e0c30cf7e606f4 WatchSource:0}: Error finding container 1ac4243caaa1ffe16e3f3e72d2d1d80bc81fbd5a8e63892b64e0c30cf7e606f4: Status 404 returned error can't find the container with id 1ac4243caaa1ffe16e3f3e72d2d1d80bc81fbd5a8e63892b64e0c30cf7e606f4
Sep 16 19:09:43.569275 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:43.569240    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" event={"ID":"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b","Type":"ContainerStarted","Data":"5703065ed9f063274d0475262fb37b035adb0e16fc4824a711980ba39e8e6705"}
Sep 16 19:09:43.569736 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:43.569308    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" event={"ID":"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b","Type":"ContainerStarted","Data":"1ac4243caaa1ffe16e3f3e72d2d1d80bc81fbd5a8e63892b64e0c30cf7e606f4"}
Sep 16 19:09:47.589898 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:47.589866    2582 generic.go:356] "Generic (PLEG): container finished" podID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerID="5703065ed9f063274d0475262fb37b035adb0e16fc4824a711980ba39e8e6705" exitCode=0
Sep 16 19:09:47.590317 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:47.589943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" event={"ID":"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b","Type":"ContainerDied","Data":"5703065ed9f063274d0475262fb37b035adb0e16fc4824a711980ba39e8e6705"}
Sep 16 19:09:48.597230 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:48.597195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" event={"ID":"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b","Type":"ContainerStarted","Data":"f07b0800476460b5a2430c181369f34df6c2d44effc94e0a87999e51e1199d22"}
Sep 16 19:09:48.597608 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:48.597286    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" event={"ID":"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b","Type":"ContainerStarted","Data":"76cc06ddc59d58fcef79e96deebcc537fe98f308b8b7cea9e4774a916c5a87e9"}
Sep 16 19:09:48.597608 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:48.597476    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:48.625549 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:48.625506    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" podStartSLOduration=2.8295636330000002 podStartE2EDuration="6.625492519s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.795928886s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:48.623726823 +0000 UTC m=+1783.811102610" watchObservedRunningTime="2026-09-16 19:09:48.625492519 +0000 UTC m=+1783.812868306"
Sep 16 19:09:49.602774 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:49.602744    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:54.361481 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.361448    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"]
Sep 16 19:09:54.364131 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.364111    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.368249 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.368228    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 16 19:09:54.368365 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.368251    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:09:54.381022 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.380997    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"]
Sep 16 19:09:54.431678 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.431656    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"]
Sep 16 19:09:54.434256 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.434242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.436612 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.436594    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:09:54.447274 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.447254    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"]
Sep 16 19:09:54.500713 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.500694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.500811 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.500733    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.500811 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.500753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.500887 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.500805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8whf\" (UniqueName: \"kubernetes.io/projected/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kube-api-access-x8whf\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.500887 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.500876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.601450 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.601426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.601553 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.601455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.601553 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.601478    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8whf\" (UniqueName: \"kubernetes.io/projected/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kube-api-access-x8whf\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.601553 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.601509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88kv8\" (UniqueName: \"kubernetes.io/projected/a4c0c228-54ac-4eed-830e-a63f293d27bd-kube-api-access-88kv8\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.601699 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.601557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c0c228-54ac-4eed-830e-a63f293d27bd-kserve-provision-location\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.601699 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.601588    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4c0c228-54ac-4eed-830e-a63f293d27bd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.601808 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.601694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.601808 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.601741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c0c228-54ac-4eed-830e-a63f293d27bd-cabundle-cert\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.601808 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.601774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4c0c228-54ac-4eed-830e-a63f293d27bd-proxy-tls\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.601999 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.601891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.602041 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:09:54.601999    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 16 19:09:54.602107 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:09:54.602095    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-proxy-tls podName:c23b2ff0-975c-40c0-abe4-5e10329ec5b3 nodeName:}" failed. No retries permitted until 2026-09-16 19:09:55.102076812 +0000 UTC m=+1790.289452579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-proxy-tls") pod "isvc-canary-create-v2-predictor-85b98c9747-6cpd7" (UID: "c23b2ff0-975c-40c0-abe4-5e10329ec5b3") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 16 19:09:54.602107 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.602089    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.602203 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.602114    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.602203 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.602163    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.613262 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.613207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8whf\" (UniqueName: \"kubernetes.io/projected/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kube-api-access-x8whf\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:54.702698 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.702670    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c0c228-54ac-4eed-830e-a63f293d27bd-cabundle-cert\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.702795 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.702703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4c0c228-54ac-4eed-830e-a63f293d27bd-proxy-tls\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.702795 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.702789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88kv8\" (UniqueName: \"kubernetes.io/projected/a4c0c228-54ac-4eed-830e-a63f293d27bd-kube-api-access-88kv8\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.702890 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.702840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c0c228-54ac-4eed-830e-a63f293d27bd-kserve-provision-location\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.702890 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.702869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4c0c228-54ac-4eed-830e-a63f293d27bd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.703236 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.703215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c0c228-54ac-4eed-830e-a63f293d27bd-kserve-provision-location\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.703480 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.703456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c0c228-54ac-4eed-830e-a63f293d27bd-cabundle-cert\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.703526 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.703484    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4c0c228-54ac-4eed-830e-a63f293d27bd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.705172 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.705152    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4c0c228-54ac-4eed-830e-a63f293d27bd-proxy-tls\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.718520 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.718500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88kv8\" (UniqueName: \"kubernetes.io/projected/a4c0c228-54ac-4eed-830e-a63f293d27bd-kube-api-access-88kv8\") pod \"isvc-canary-create-predictor-76cc455874-skjsj\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:54.743756 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:54.743731    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:09:55.021223 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:55.021199    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"]
Sep 16 19:09:55.022884 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:09:55.022857    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c0c228_54ac_4eed_830e_a63f293d27bd.slice/crio-8a28d6ceebe2fc91fcc5cff058fe5b4a20a00bafa7c707cfa85a0c890e64f421 WatchSource:0}: Error finding container 8a28d6ceebe2fc91fcc5cff058fe5b4a20a00bafa7c707cfa85a0c890e64f421: Status 404 returned error can't find the container with id 8a28d6ceebe2fc91fcc5cff058fe5b4a20a00bafa7c707cfa85a0c890e64f421
Sep 16 19:09:55.106322 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:55.106289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:55.106430 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:09:55.106415    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 16 19:09:55.106480 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:09:55.106466    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-proxy-tls podName:c23b2ff0-975c-40c0-abe4-5e10329ec5b3 nodeName:}" failed. No retries permitted until 2026-09-16 19:09:56.106451471 +0000 UTC m=+1791.293827236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-proxy-tls") pod "isvc-canary-create-v2-predictor-85b98c9747-6cpd7" (UID: "c23b2ff0-975c-40c0-abe4-5e10329ec5b3") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 16 19:09:55.612712 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:55.612685    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:09:55.631452 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:55.631424    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" event={"ID":"a4c0c228-54ac-4eed-830e-a63f293d27bd","Type":"ContainerStarted","Data":"aa7c35b096f0fe6b65f2827f361bc5109358c44d9ae8053bc2e9776a19ed5885"}
Sep 16 19:09:55.631584 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:55.631474    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" event={"ID":"a4c0c228-54ac-4eed-830e-a63f293d27bd","Type":"ContainerStarted","Data":"8a28d6ceebe2fc91fcc5cff058fe5b4a20a00bafa7c707cfa85a0c890e64f421"}
Sep 16 19:09:56.112764 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:56.112733    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:56.115240 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:56.115215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-proxy-tls\") pod \"isvc-canary-create-v2-predictor-85b98c9747-6cpd7\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:56.173940 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:56.173914    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:09:56.448209 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:56.448185    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"]
Sep 16 19:09:56.450359 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:09:56.450311    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23b2ff0_975c_40c0_abe4_5e10329ec5b3.slice/crio-db79a8da18d99a4ff7efd2fd1919ce63f1231ce6902f986be7bc3becc1c3db3a WatchSource:0}: Error finding container db79a8da18d99a4ff7efd2fd1919ce63f1231ce6902f986be7bc3becc1c3db3a: Status 404 returned error can't find the container with id db79a8da18d99a4ff7efd2fd1919ce63f1231ce6902f986be7bc3becc1c3db3a
Sep 16 19:09:56.638601 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:56.638568    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" event={"ID":"c23b2ff0-975c-40c0-abe4-5e10329ec5b3","Type":"ContainerStarted","Data":"604a7cdbc83673de34c234785deacd648d813b27aceb5b33e135a9536efd7400"}
Sep 16 19:09:56.639040 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:56.638621    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" event={"ID":"c23b2ff0-975c-40c0-abe4-5e10329ec5b3","Type":"ContainerStarted","Data":"db79a8da18d99a4ff7efd2fd1919ce63f1231ce6902f986be7bc3becc1c3db3a"}
Sep 16 19:09:59.654587 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:59.654557    2582 generic.go:356] "Generic (PLEG): container finished" podID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerID="aa7c35b096f0fe6b65f2827f361bc5109358c44d9ae8053bc2e9776a19ed5885" exitCode=0
Sep 16 19:09:59.655003 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:09:59.654632    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" event={"ID":"a4c0c228-54ac-4eed-830e-a63f293d27bd","Type":"ContainerDied","Data":"aa7c35b096f0fe6b65f2827f361bc5109358c44d9ae8053bc2e9776a19ed5885"}
Sep 16 19:10:00.660844 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:00.660771    2582 generic.go:356] "Generic (PLEG): container finished" podID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerID="604a7cdbc83673de34c234785deacd648d813b27aceb5b33e135a9536efd7400" exitCode=0
Sep 16 19:10:00.661280 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:00.660843    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" event={"ID":"c23b2ff0-975c-40c0-abe4-5e10329ec5b3","Type":"ContainerDied","Data":"604a7cdbc83673de34c234785deacd648d813b27aceb5b33e135a9536efd7400"}
Sep 16 19:10:00.663399 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:00.663366    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" event={"ID":"a4c0c228-54ac-4eed-830e-a63f293d27bd","Type":"ContainerStarted","Data":"d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7"}
Sep 16 19:10:00.663501 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:00.663438    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" event={"ID":"a4c0c228-54ac-4eed-830e-a63f293d27bd","Type":"ContainerStarted","Data":"03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a"}
Sep 16 19:10:00.663637 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:00.663621    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:10:00.709987 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:00.709859    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podStartSLOduration=2.8101847639999997 podStartE2EDuration="6.709841175s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.899656411s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:00.709434227 +0000 UTC m=+1795.896810015" watchObservedRunningTime="2026-09-16 19:10:00.709841175 +0000 UTC m=+1795.897216979"
Sep 16 19:10:01.671563 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:01.671527    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" event={"ID":"c23b2ff0-975c-40c0-abe4-5e10329ec5b3","Type":"ContainerStarted","Data":"d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff"}
Sep 16 19:10:01.672017 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:01.671627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" event={"ID":"c23b2ff0-975c-40c0-abe4-5e10329ec5b3","Type":"ContainerStarted","Data":"d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039"}
Sep 16 19:10:01.672017 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:01.671664    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:10:01.672017 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:01.671844    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:10:01.672894 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:01.672872    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:10:01.695716 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:01.695674    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" podStartSLOduration=3.898140916 podStartE2EDuration="7.695659409s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.797518493s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:01.694686933 +0000 UTC m=+1796.882062721" watchObservedRunningTime="2026-09-16 19:10:01.695659409 +0000 UTC m=+1796.883035196"
Sep 16 19:10:02.677118 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:02.677071    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:10:02.677561 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:02.677245    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:10:02.678702 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:02.678675    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 16 19:10:03.682221 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:03.682184    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 16 19:10:05.354213 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:05.354186    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:10:05.360639 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:05.360615    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:10:07.681727 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:07.681694    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:10:07.682285 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:07.682256    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:10:08.686928 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:08.686886    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:10:08.687521 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:08.687498    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 16 19:10:17.683056 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:17.683008    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:10:18.688498 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:18.688458    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 16 19:10:25.618378 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:25.618329    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:10:27.682854 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:27.682816    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:10:28.687711 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:28.687672    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 16 19:10:35.615340 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:35.615314    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:10:37.683157 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:37.683117    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:10:38.688237 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:38.688197    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 16 19:10:42.554938 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:42.554896    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"]
Sep 16 19:10:42.555346 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:42.555168    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="kserve-container" containerID="cri-o://76cc06ddc59d58fcef79e96deebcc537fe98f308b8b7cea9e4774a916c5a87e9" gracePeriod=30
Sep 16 19:10:42.555346 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:42.555188    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="kube-rbac-proxy" containerID="cri-o://f07b0800476460b5a2430c181369f34df6c2d44effc94e0a87999e51e1199d22" gracePeriod=30
Sep 16 19:10:42.869217 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:42.869127    2582 generic.go:356] "Generic (PLEG): container finished" podID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerID="f07b0800476460b5a2430c181369f34df6c2d44effc94e0a87999e51e1199d22" exitCode=2
Sep 16 19:10:42.869217 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:42.869196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" event={"ID":"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b","Type":"ContainerDied","Data":"f07b0800476460b5a2430c181369f34df6c2d44effc94e0a87999e51e1199d22"}
Sep 16 19:10:45.608342 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:45.608301    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 16 19:10:45.613771 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:45.613745    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.40:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 16 19:10:47.683073 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:47.683033    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:10:48.688191 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:48.688147    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 16 19:10:48.900493 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:48.900461    2582 generic.go:356] "Generic (PLEG): container finished" podID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerID="76cc06ddc59d58fcef79e96deebcc537fe98f308b8b7cea9e4774a916c5a87e9" exitCode=0
Sep 16 19:10:48.900640 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:48.900492    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" event={"ID":"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b","Type":"ContainerDied","Data":"76cc06ddc59d58fcef79e96deebcc537fe98f308b8b7cea9e4774a916c5a87e9"}
Sep 16 19:10:48.900640 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:48.900585    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd" event={"ID":"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b","Type":"ContainerDied","Data":"1ac4243caaa1ffe16e3f3e72d2d1d80bc81fbd5a8e63892b64e0c30cf7e606f4"}
Sep 16 19:10:48.900640 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:48.900612    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ac4243caaa1ffe16e3f3e72d2d1d80bc81fbd5a8e63892b64e0c30cf7e606f4"
Sep 16 19:10:48.902332 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:48.902315    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:10:49.053297 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.053226    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-proxy-tls\") pod \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") "
Sep 16 19:10:49.053297 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.053289    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-cabundle-cert\") pod \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") "
Sep 16 19:10:49.053513 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.053317    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kserve-provision-location\") pod \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") "
Sep 16 19:10:49.053513 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.053341    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxc76\" (UniqueName: \"kubernetes.io/projected/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kube-api-access-rxc76\") pod \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") "
Sep 16 19:10:49.053513 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.053365    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kube-rbac-proxy-sar-config\") pod \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\" (UID: \"8a1de4a9-74b4-4a83-ad74-efca0fe02f8b\") "
Sep 16 19:10:49.053689 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.053650    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" (UID: "8a1de4a9-74b4-4a83-ad74-efca0fe02f8b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:49.053689 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.053681    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" (UID: "8a1de4a9-74b4-4a83-ad74-efca0fe02f8b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:49.053797 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.053740    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" (UID: "8a1de4a9-74b4-4a83-ad74-efca0fe02f8b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:49.055629 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.055593    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" (UID: "8a1de4a9-74b4-4a83-ad74-efca0fe02f8b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:49.055723 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.055643    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kube-api-access-rxc76" (OuterVolumeSpecName: "kube-api-access-rxc76") pod "8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" (UID: "8a1de4a9-74b4-4a83-ad74-efca0fe02f8b"). InnerVolumeSpecName "kube-api-access-rxc76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:49.154240 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.154208    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:10:49.154240 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.154241    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:10:49.154240 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.154251    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:10:49.154413 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.154260    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxc76\" (UniqueName: \"kubernetes.io/projected/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kube-api-access-rxc76\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:10:49.154413 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.154269    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:10:49.905825 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.905793    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"
Sep 16 19:10:49.976989 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:49.976958    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"]
Sep 16 19:10:50.006775 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:50.006746    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd"]
Sep 16 19:10:51.296891 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:51.296855    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" path="/var/lib/kubelet/pods/8a1de4a9-74b4-4a83-ad74-efca0fe02f8b/volumes"
Sep 16 19:10:57.682689 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:57.682656    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:10:58.688010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:10:58.687977    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 16 19:11:07.682942 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:07.682885    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:11:08.688295 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:08.688255    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 16 19:11:17.682828 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:17.682802    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:11:18.688057 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:18.688032    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:11:24.429972 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:24.429900    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"]
Sep 16 19:11:24.430367 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:24.430158    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container" containerID="cri-o://03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a" gracePeriod=30
Sep 16 19:11:24.430367 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:24.430202    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kube-rbac-proxy" containerID="cri-o://d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7" gracePeriod=30
Sep 16 19:11:24.437615 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:24.437588    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"]
Sep 16 19:11:24.437822 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:24.437788    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container" containerID="cri-o://d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039" gracePeriod=30
Sep 16 19:11:24.437924 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:24.437874    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kube-rbac-proxy" containerID="cri-o://d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff" gracePeriod=30
Sep 16 19:11:25.082416 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:25.082382    2582 generic.go:356] "Generic (PLEG): container finished" podID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerID="d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff" exitCode=2
Sep 16 19:11:25.082557 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:25.082456    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" event={"ID":"c23b2ff0-975c-40c0-abe4-5e10329ec5b3","Type":"ContainerDied","Data":"d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff"}
Sep 16 19:11:25.084644 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:25.084621    2582 generic.go:356] "Generic (PLEG): container finished" podID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerID="d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7" exitCode=2
Sep 16 19:11:25.084758 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:25.084654    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" event={"ID":"a4c0c228-54ac-4eed-830e-a63f293d27bd","Type":"ContainerDied","Data":"d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7"}
Sep 16 19:11:27.678141 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:27.678046    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 16 19:11:27.682381 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:27.682353    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:11:28.477167 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.477144    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:11:28.551988 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.551966    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88kv8\" (UniqueName: \"kubernetes.io/projected/a4c0c228-54ac-4eed-830e-a63f293d27bd-kube-api-access-88kv8\") pod \"a4c0c228-54ac-4eed-830e-a63f293d27bd\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") "
Sep 16 19:11:28.552090 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.552002    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4c0c228-54ac-4eed-830e-a63f293d27bd-proxy-tls\") pod \"a4c0c228-54ac-4eed-830e-a63f293d27bd\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") "
Sep 16 19:11:28.552090 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.552028    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c0c228-54ac-4eed-830e-a63f293d27bd-kserve-provision-location\") pod \"a4c0c228-54ac-4eed-830e-a63f293d27bd\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") "
Sep 16 19:11:28.552172 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.552130    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4c0c228-54ac-4eed-830e-a63f293d27bd-kube-rbac-proxy-sar-config\") pod \"a4c0c228-54ac-4eed-830e-a63f293d27bd\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") "
Sep 16 19:11:28.552222 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.552182    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c0c228-54ac-4eed-830e-a63f293d27bd-cabundle-cert\") pod \"a4c0c228-54ac-4eed-830e-a63f293d27bd\" (UID: \"a4c0c228-54ac-4eed-830e-a63f293d27bd\") "
Sep 16 19:11:28.552394 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.552367    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c0c228-54ac-4eed-830e-a63f293d27bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4c0c228-54ac-4eed-830e-a63f293d27bd" (UID: "a4c0c228-54ac-4eed-830e-a63f293d27bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:28.552538 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.552509    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c0c228-54ac-4eed-830e-a63f293d27bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4c0c228-54ac-4eed-830e-a63f293d27bd" (UID: "a4c0c228-54ac-4eed-830e-a63f293d27bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:28.552538 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.552517    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c0c228-54ac-4eed-830e-a63f293d27bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4c0c228-54ac-4eed-830e-a63f293d27bd" (UID: "a4c0c228-54ac-4eed-830e-a63f293d27bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:28.554106 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.554085    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c0c228-54ac-4eed-830e-a63f293d27bd-kube-api-access-88kv8" (OuterVolumeSpecName: "kube-api-access-88kv8") pod "a4c0c228-54ac-4eed-830e-a63f293d27bd" (UID: "a4c0c228-54ac-4eed-830e-a63f293d27bd"). InnerVolumeSpecName "kube-api-access-88kv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:28.554537 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.554521    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c0c228-54ac-4eed-830e-a63f293d27bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4c0c228-54ac-4eed-830e-a63f293d27bd" (UID: "a4c0c228-54ac-4eed-830e-a63f293d27bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:28.653642 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.653609    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4c0c228-54ac-4eed-830e-a63f293d27bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:11:28.653642 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.653638    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c0c228-54ac-4eed-830e-a63f293d27bd-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:11:28.653642 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.653648    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88kv8\" (UniqueName: \"kubernetes.io/projected/a4c0c228-54ac-4eed-830e-a63f293d27bd-kube-api-access-88kv8\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:11:28.653868 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.653657    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4c0c228-54ac-4eed-830e-a63f293d27bd-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:11:28.653868 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.653667    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c0c228-54ac-4eed-830e-a63f293d27bd-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:11:28.674625 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.674603    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:11:28.754353 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.754324    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8whf\" (UniqueName: \"kubernetes.io/projected/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kube-api-access-x8whf\") pod \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") "
Sep 16 19:11:28.754753 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.754383    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-proxy-tls\") pod \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") "
Sep 16 19:11:28.754753 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.754416    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kube-rbac-proxy-sar-config\") pod \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") "
Sep 16 19:11:28.754753 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.754431    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-cabundle-cert\") pod \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") "
Sep 16 19:11:28.754753 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.754577    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kserve-provision-location\") pod \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\" (UID: \"c23b2ff0-975c-40c0-abe4-5e10329ec5b3\") "
Sep 16 19:11:28.754983 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.754819    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c23b2ff0-975c-40c0-abe4-5e10329ec5b3" (UID: "c23b2ff0-975c-40c0-abe4-5e10329ec5b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:28.754983 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.754845    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c23b2ff0-975c-40c0-abe4-5e10329ec5b3" (UID: "c23b2ff0-975c-40c0-abe4-5e10329ec5b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:28.754983 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.754889    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c23b2ff0-975c-40c0-abe4-5e10329ec5b3" (UID: "c23b2ff0-975c-40c0-abe4-5e10329ec5b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:28.756600 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.756577    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kube-api-access-x8whf" (OuterVolumeSpecName: "kube-api-access-x8whf") pod "c23b2ff0-975c-40c0-abe4-5e10329ec5b3" (UID: "c23b2ff0-975c-40c0-abe4-5e10329ec5b3"). InnerVolumeSpecName "kube-api-access-x8whf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:28.756600 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.756592    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c23b2ff0-975c-40c0-abe4-5e10329ec5b3" (UID: "c23b2ff0-975c-40c0-abe4-5e10329ec5b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:28.856142 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.856091    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:11:28.856142 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.856113    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8whf\" (UniqueName: \"kubernetes.io/projected/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kube-api-access-x8whf\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:11:28.856142 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.856123    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:11:28.856142 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.856132    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:11:28.856142 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:28.856140    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c23b2ff0-975c-40c0-abe4-5e10329ec5b3-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:11:29.107828 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.107744    2582 generic.go:356] "Generic (PLEG): container finished" podID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerID="d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039" exitCode=0
Sep 16 19:11:29.107828 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.107808    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"
Sep 16 19:11:29.108052 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.107834    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" event={"ID":"c23b2ff0-975c-40c0-abe4-5e10329ec5b3","Type":"ContainerDied","Data":"d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039"}
Sep 16 19:11:29.108052 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.107929    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7" event={"ID":"c23b2ff0-975c-40c0-abe4-5e10329ec5b3","Type":"ContainerDied","Data":"db79a8da18d99a4ff7efd2fd1919ce63f1231ce6902f986be7bc3becc1c3db3a"}
Sep 16 19:11:29.108052 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.107952    2582 scope.go:117] "RemoveContainer" containerID="d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff"
Sep 16 19:11:29.110105 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.110081    2582 generic.go:356] "Generic (PLEG): container finished" podID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerID="03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a" exitCode=0
Sep 16 19:11:29.110218 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.110141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" event={"ID":"a4c0c228-54ac-4eed-830e-a63f293d27bd","Type":"ContainerDied","Data":"03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a"}
Sep 16 19:11:29.110218 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.110150    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"
Sep 16 19:11:29.110218 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.110183    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj" event={"ID":"a4c0c228-54ac-4eed-830e-a63f293d27bd","Type":"ContainerDied","Data":"8a28d6ceebe2fc91fcc5cff058fe5b4a20a00bafa7c707cfa85a0c890e64f421"}
Sep 16 19:11:29.117949 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.117926    2582 scope.go:117] "RemoveContainer" containerID="d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039"
Sep 16 19:11:29.126188 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.126169    2582 scope.go:117] "RemoveContainer" containerID="604a7cdbc83673de34c234785deacd648d813b27aceb5b33e135a9536efd7400"
Sep 16 19:11:29.133585 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.133570    2582 scope.go:117] "RemoveContainer" containerID="d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff"
Sep 16 19:11:29.133869 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:11:29.133849    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff\": container with ID starting with d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff not found: ID does not exist" containerID="d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff"
Sep 16 19:11:29.133973 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.133875    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff"} err="failed to get container status \"d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff\": rpc error: code = NotFound desc = could not find container \"d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff\": container with ID starting with d3982f38711054039b445c06ac47caff0e4ef1c5f452991d45a2c8dcda87e1ff not found: ID does not exist"
Sep 16 19:11:29.133973 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.133892    2582 scope.go:117] "RemoveContainer" containerID="d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039"
Sep 16 19:11:29.134219 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:11:29.134198    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039\": container with ID starting with d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039 not found: ID does not exist" containerID="d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039"
Sep 16 19:11:29.134301 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.134227    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039"} err="failed to get container status \"d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039\": rpc error: code = NotFound desc = could not find container \"d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039\": container with ID starting with d850b1bd9e505793bae0925ec9bf5c81cf12d3bef66b36a77034c814561e0039 not found: ID does not exist"
Sep 16 19:11:29.134301 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.134247    2582 scope.go:117] "RemoveContainer" containerID="604a7cdbc83673de34c234785deacd648d813b27aceb5b33e135a9536efd7400"
Sep 16 19:11:29.134519 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:11:29.134502    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604a7cdbc83673de34c234785deacd648d813b27aceb5b33e135a9536efd7400\": container with ID starting with 604a7cdbc83673de34c234785deacd648d813b27aceb5b33e135a9536efd7400 not found: ID does not exist" containerID="604a7cdbc83673de34c234785deacd648d813b27aceb5b33e135a9536efd7400"
Sep 16 19:11:29.134569 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.134523    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604a7cdbc83673de34c234785deacd648d813b27aceb5b33e135a9536efd7400"} err="failed to get container status \"604a7cdbc83673de34c234785deacd648d813b27aceb5b33e135a9536efd7400\": rpc error: code = NotFound desc = could not find container \"604a7cdbc83673de34c234785deacd648d813b27aceb5b33e135a9536efd7400\": container with ID starting with 604a7cdbc83673de34c234785deacd648d813b27aceb5b33e135a9536efd7400 not found: ID does not exist"
Sep 16 19:11:29.134569 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.134542    2582 scope.go:117] "RemoveContainer" containerID="d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7"
Sep 16 19:11:29.141728 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.141712    2582 scope.go:117] "RemoveContainer" containerID="03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a"
Sep 16 19:11:29.148609 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.148592    2582 scope.go:117] "RemoveContainer" containerID="aa7c35b096f0fe6b65f2827f361bc5109358c44d9ae8053bc2e9776a19ed5885"
Sep 16 19:11:29.155851 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.155810    2582 scope.go:117] "RemoveContainer" containerID="d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7"
Sep 16 19:11:29.156458 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:11:29.156438    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7\": container with ID starting with d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7 not found: ID does not exist" containerID="d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7"
Sep 16 19:11:29.156538 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.156462    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7"} err="failed to get container status \"d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7\": rpc error: code = NotFound desc = could not find container \"d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7\": container with ID starting with d42e3fd3f1465284a9ff71dadd19d455fde44f7b9d3151b675190f9b23251ab7 not found: ID does not exist"
Sep 16 19:11:29.156538 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.156479    2582 scope.go:117] "RemoveContainer" containerID="03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a"
Sep 16 19:11:29.156765 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:11:29.156748    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a\": container with ID starting with 03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a not found: ID does not exist" containerID="03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a"
Sep 16 19:11:29.156831 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.156773    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a"} err="failed to get container status \"03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a\": rpc error: code = NotFound desc = could not find container \"03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a\": container with ID starting with 03a4af4489f91c235b5b15648be81a0b7eca1cf3d25ff78f21b0aa646883a40a not found: ID does not exist"
Sep 16 19:11:29.156831 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.156796    2582 scope.go:117] "RemoveContainer" containerID="aa7c35b096f0fe6b65f2827f361bc5109358c44d9ae8053bc2e9776a19ed5885"
Sep 16 19:11:29.157108 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:11:29.157085    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7c35b096f0fe6b65f2827f361bc5109358c44d9ae8053bc2e9776a19ed5885\": container with ID starting with aa7c35b096f0fe6b65f2827f361bc5109358c44d9ae8053bc2e9776a19ed5885 not found: ID does not exist" containerID="aa7c35b096f0fe6b65f2827f361bc5109358c44d9ae8053bc2e9776a19ed5885"
Sep 16 19:11:29.157211 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.157112    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7c35b096f0fe6b65f2827f361bc5109358c44d9ae8053bc2e9776a19ed5885"} err="failed to get container status \"aa7c35b096f0fe6b65f2827f361bc5109358c44d9ae8053bc2e9776a19ed5885\": rpc error: code = NotFound desc = could not find container \"aa7c35b096f0fe6b65f2827f361bc5109358c44d9ae8053bc2e9776a19ed5885\": container with ID starting with aa7c35b096f0fe6b65f2827f361bc5109358c44d9ae8053bc2e9776a19ed5885 not found: ID does not exist"
Sep 16 19:11:29.158089 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.158065    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"]
Sep 16 19:11:29.160044 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.160024    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7"]
Sep 16 19:11:29.176235 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.176206    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"]
Sep 16 19:11:29.178831 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.178810    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj"]
Sep 16 19:11:29.296887 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.296858    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" path="/var/lib/kubelet/pods/a4c0c228-54ac-4eed-830e-a63f293d27bd/volumes"
Sep 16 19:11:29.297365 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.297351    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" path="/var/lib/kubelet/pods/c23b2ff0-975c-40c0-abe4-5e10329ec5b3/volumes"
Sep 16 19:11:29.780066 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780033    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"]
Sep 16 19:11:29.780440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780363    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="kserve-container"
Sep 16 19:11:29.780440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780373    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="kserve-container"
Sep 16 19:11:29.780440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780382    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="storage-initializer"
Sep 16 19:11:29.780440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780388    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="storage-initializer"
Sep 16 19:11:29.780440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780395    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container"
Sep 16 19:11:29.780440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780401    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container"
Sep 16 19:11:29.780440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780407    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kube-rbac-proxy"
Sep 16 19:11:29.780440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780413    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kube-rbac-proxy"
Sep 16 19:11:29.780440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780419    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container"
Sep 16 19:11:29.780440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780424    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container"
Sep 16 19:11:29.780440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780439    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kube-rbac-proxy"
Sep 16 19:11:29.780440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780444    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kube-rbac-proxy"
Sep 16 19:11:29.781103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780450    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="storage-initializer"
Sep 16 19:11:29.781103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780455    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="storage-initializer"
Sep 16 19:11:29.781103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780463    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="kube-rbac-proxy"
Sep 16 19:11:29.781103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780468    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="kube-rbac-proxy"
Sep 16 19:11:29.781103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780477    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="storage-initializer"
Sep 16 19:11:29.781103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780482    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="storage-initializer"
Sep 16 19:11:29.781103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780528    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kserve-container"
Sep 16 19:11:29.781103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780536    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kube-rbac-proxy"
Sep 16 19:11:29.781103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780541    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c23b2ff0-975c-40c0-abe4-5e10329ec5b3" containerName="kserve-container"
Sep 16 19:11:29.781103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780548    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="kube-rbac-proxy"
Sep 16 19:11:29.781103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780553    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4c0c228-54ac-4eed-830e-a63f293d27bd" containerName="kube-rbac-proxy"
Sep 16 19:11:29.781103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.780559    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1de4a9-74b4-4a83-ad74-efca0fe02f8b" containerName="kserve-container"
Sep 16 19:11:29.785522 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.785505    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.788713 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.788693    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:11:29.788937 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.788922    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:11:29.789262 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.789243    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:11:29.789471 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.789449    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:11:29.789471 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.789462    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:11:29.789616 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.789478    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:11:29.790594 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.790581    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:11:29.796359 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.796339    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"]
Sep 16 19:11:29.863960 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.863934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb4dj\" (UniqueName: \"kubernetes.io/projected/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kube-api-access-fb4dj\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.864070 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.863978    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.864139 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.864074    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e25bb26a-18c2-4b40-9fff-2e8f21199c06-proxy-tls\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.864139 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.864104    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e25bb26a-18c2-4b40-9fff-2e8f21199c06-cabundle-cert\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.864245 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.864176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.964977 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.964938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e25bb26a-18c2-4b40-9fff-2e8f21199c06-proxy-tls\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.964977 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.964981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e25bb26a-18c2-4b40-9fff-2e8f21199c06-cabundle-cert\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.965182 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.965023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.965182 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.965058    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb4dj\" (UniqueName: \"kubernetes.io/projected/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kube-api-access-fb4dj\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.965182 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.965092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.965579 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.965559    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.965727 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.965708    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e25bb26a-18c2-4b40-9fff-2e8f21199c06-cabundle-cert\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.965772 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.965723    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.967569 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.967552    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e25bb26a-18c2-4b40-9fff-2e8f21199c06-proxy-tls\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:29.975870 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:29.975846    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb4dj\" (UniqueName: \"kubernetes.io/projected/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kube-api-access-fb4dj\") pod \"isvc-canary-promote-predictor-6b6f66ff94-cn57m\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:30.096584 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:30.096512    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:30.383360 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:30.383290    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"]
Sep 16 19:11:30.386088 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:11:30.386048    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25bb26a_18c2_4b40_9fff_2e8f21199c06.slice/crio-9d0ebd21e909f94ba8a9f5f052307ece20a9e6e25208a2e08ad2c6384ccc5d18 WatchSource:0}: Error finding container 9d0ebd21e909f94ba8a9f5f052307ece20a9e6e25208a2e08ad2c6384ccc5d18: Status 404 returned error can't find the container with id 9d0ebd21e909f94ba8a9f5f052307ece20a9e6e25208a2e08ad2c6384ccc5d18
Sep 16 19:11:31.123223 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:31.123182    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" event={"ID":"e25bb26a-18c2-4b40-9fff-2e8f21199c06","Type":"ContainerStarted","Data":"987a35905187188345af6fe3600bf49b075e503b74022a66c6258962664375b5"}
Sep 16 19:11:31.123639 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:31.123296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" event={"ID":"e25bb26a-18c2-4b40-9fff-2e8f21199c06","Type":"ContainerStarted","Data":"9d0ebd21e909f94ba8a9f5f052307ece20a9e6e25208a2e08ad2c6384ccc5d18"}
Sep 16 19:11:35.144737 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:35.144706    2582 generic.go:356] "Generic (PLEG): container finished" podID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerID="987a35905187188345af6fe3600bf49b075e503b74022a66c6258962664375b5" exitCode=0
Sep 16 19:11:35.145143 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:35.144782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" event={"ID":"e25bb26a-18c2-4b40-9fff-2e8f21199c06","Type":"ContainerDied","Data":"987a35905187188345af6fe3600bf49b075e503b74022a66c6258962664375b5"}
Sep 16 19:11:36.151977 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:36.151944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" event={"ID":"e25bb26a-18c2-4b40-9fff-2e8f21199c06","Type":"ContainerStarted","Data":"1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f"}
Sep 16 19:11:36.152343 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:36.152031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" event={"ID":"e25bb26a-18c2-4b40-9fff-2e8f21199c06","Type":"ContainerStarted","Data":"42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d"}
Sep 16 19:11:36.152343 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:36.152210    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:36.187521 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:36.187475    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podStartSLOduration=3.401677113 podStartE2EDuration="7.187461292s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.785784179s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:36.185514835 +0000 UTC m=+1891.372890621" watchObservedRunningTime="2026-09-16 19:11:36.187461292 +0000 UTC m=+1891.374837078"
Sep 16 19:11:37.157135 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:37.157104    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:37.158208 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:37.158181    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:11:38.161947 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:38.161887    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:11:43.167079 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:43.167050    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:11:43.167579 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:43.167548    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:11:53.168336 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:11:53.168294    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:12:03.167812 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:12:03.167770    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:12:13.168118 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:12:13.168078    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:12:23.167510 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:12:23.167471    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:12:33.168178 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:12:33.168140    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:12:43.167768 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:12:43.167724    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:12:53.168936 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:12:53.168864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:13:04.912923 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:04.912807    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"]
Sep 16 19:13:04.913434 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:04.913107    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container" containerID="cri-o://42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d" gracePeriod=30
Sep 16 19:13:04.913434 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:04.913220    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kube-rbac-proxy" containerID="cri-o://1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f" gracePeriod=30
Sep 16 19:13:05.060035 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.060006    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"]
Sep 16 19:13:05.063308 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.063293    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.079818 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.079799    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:13:05.094134 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.094116    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"]
Sep 16 19:13:05.176788 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.176764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc73837a-ce73-45f1-b2d8-fd3df2229464-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.176889 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.176799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc73837a-ce73-45f1-b2d8-fd3df2229464-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.176889 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.176821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhwn8\" (UniqueName: \"kubernetes.io/projected/bc73837a-ce73-45f1-b2d8-fd3df2229464-kube-api-access-rhwn8\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.176889 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.176859    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc73837a-ce73-45f1-b2d8-fd3df2229464-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.177040 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.176953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc73837a-ce73-45f1-b2d8-fd3df2229464-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.278242 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.278209    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc73837a-ce73-45f1-b2d8-fd3df2229464-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.278339 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.278250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc73837a-ce73-45f1-b2d8-fd3df2229464-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.278339 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.278275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwn8\" (UniqueName: \"kubernetes.io/projected/bc73837a-ce73-45f1-b2d8-fd3df2229464-kube-api-access-rhwn8\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.278339 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.278301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc73837a-ce73-45f1-b2d8-fd3df2229464-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.278446 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.278344    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc73837a-ce73-45f1-b2d8-fd3df2229464-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.278657 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.278637    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc73837a-ce73-45f1-b2d8-fd3df2229464-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.278900 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.278884    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc73837a-ce73-45f1-b2d8-fd3df2229464-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.278997 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.278977    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc73837a-ce73-45f1-b2d8-fd3df2229464-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.284488 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.284472    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:13:05.290935 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.290898    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc73837a-ce73-45f1-b2d8-fd3df2229464-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.295594 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.295573    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwn8\" (UniqueName: \"kubernetes.io/projected/bc73837a-ce73-45f1-b2d8-fd3df2229464-kube-api-access-rhwn8\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.372843 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.372821    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:05.591592 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.591521    2582 generic.go:356] "Generic (PLEG): container finished" podID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerID="1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f" exitCode=2
Sep 16 19:13:05.591717 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.591598    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" event={"ID":"e25bb26a-18c2-4b40-9fff-2e8f21199c06","Type":"ContainerDied","Data":"1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f"}
Sep 16 19:13:05.665043 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:05.665022    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"]
Sep 16 19:13:05.667433 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:13:05.667410    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc73837a_ce73_45f1_b2d8_fd3df2229464.slice/crio-f5c184bcc20c4bd406b5c3ff84c2612188d137cabac0ee49e143a252632a7435 WatchSource:0}: Error finding container f5c184bcc20c4bd406b5c3ff84c2612188d137cabac0ee49e143a252632a7435: Status 404 returned error can't find the container with id f5c184bcc20c4bd406b5c3ff84c2612188d137cabac0ee49e143a252632a7435
Sep 16 19:13:06.599755 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:06.599719    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" event={"ID":"bc73837a-ce73-45f1-b2d8-fd3df2229464","Type":"ContainerStarted","Data":"9cc0877cc3582b3ae94dc31c70a43872d255ff98cd99230da0ce86917582d893"}
Sep 16 19:13:06.600239 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:06.599786    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" event={"ID":"bc73837a-ce73-45f1-b2d8-fd3df2229464","Type":"ContainerStarted","Data":"f5c184bcc20c4bd406b5c3ff84c2612188d137cabac0ee49e143a252632a7435"}
Sep 16 19:13:08.162480 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:08.162439    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 16 19:13:08.950111 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:08.950089    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:13:09.110992 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.110929    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e25bb26a-18c2-4b40-9fff-2e8f21199c06-proxy-tls\") pod \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") "
Sep 16 19:13:09.110992 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.110961    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kserve-provision-location\") pod \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") "
Sep 16 19:13:09.110992 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.110988    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb4dj\" (UniqueName: \"kubernetes.io/projected/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kube-api-access-fb4dj\") pod \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") "
Sep 16 19:13:09.111254 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.111006    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kube-rbac-proxy-sar-config\") pod \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") "
Sep 16 19:13:09.111254 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.111039    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e25bb26a-18c2-4b40-9fff-2e8f21199c06-cabundle-cert\") pod \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\" (UID: \"e25bb26a-18c2-4b40-9fff-2e8f21199c06\") "
Sep 16 19:13:09.111375 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.111335    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e25bb26a-18c2-4b40-9fff-2e8f21199c06" (UID: "e25bb26a-18c2-4b40-9fff-2e8f21199c06"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:09.111430 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.111396    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e25bb26a-18c2-4b40-9fff-2e8f21199c06" (UID: "e25bb26a-18c2-4b40-9fff-2e8f21199c06"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:09.111483 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.111435    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25bb26a-18c2-4b40-9fff-2e8f21199c06-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e25bb26a-18c2-4b40-9fff-2e8f21199c06" (UID: "e25bb26a-18c2-4b40-9fff-2e8f21199c06"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:09.113153 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.113135    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e25bb26a-18c2-4b40-9fff-2e8f21199c06-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e25bb26a-18c2-4b40-9fff-2e8f21199c06" (UID: "e25bb26a-18c2-4b40-9fff-2e8f21199c06"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:09.113350 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.113335    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kube-api-access-fb4dj" (OuterVolumeSpecName: "kube-api-access-fb4dj") pod "e25bb26a-18c2-4b40-9fff-2e8f21199c06" (UID: "e25bb26a-18c2-4b40-9fff-2e8f21199c06"). InnerVolumeSpecName "kube-api-access-fb4dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:09.212229 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.212201    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb4dj\" (UniqueName: \"kubernetes.io/projected/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kube-api-access-fb4dj\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:13:09.212229 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.212224    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:13:09.212561 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.212233    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e25bb26a-18c2-4b40-9fff-2e8f21199c06-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:13:09.212561 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.212243    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e25bb26a-18c2-4b40-9fff-2e8f21199c06-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:13:09.212561 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.212251    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e25bb26a-18c2-4b40-9fff-2e8f21199c06-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:13:09.615630 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.615596    2582 generic.go:356] "Generic (PLEG): container finished" podID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerID="42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d" exitCode=0
Sep 16 19:13:09.615767 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.615651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" event={"ID":"e25bb26a-18c2-4b40-9fff-2e8f21199c06","Type":"ContainerDied","Data":"42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d"}
Sep 16 19:13:09.615767 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.615741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m" event={"ID":"e25bb26a-18c2-4b40-9fff-2e8f21199c06","Type":"ContainerDied","Data":"9d0ebd21e909f94ba8a9f5f052307ece20a9e6e25208a2e08ad2c6384ccc5d18"}
Sep 16 19:13:09.615767 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.615667    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"
Sep 16 19:13:09.616002 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.615781    2582 scope.go:117] "RemoveContainer" containerID="1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f"
Sep 16 19:13:09.617760 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.617733    2582 generic.go:356] "Generic (PLEG): container finished" podID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerID="9cc0877cc3582b3ae94dc31c70a43872d255ff98cd99230da0ce86917582d893" exitCode=0
Sep 16 19:13:09.617887 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.617814    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" event={"ID":"bc73837a-ce73-45f1-b2d8-fd3df2229464","Type":"ContainerDied","Data":"9cc0877cc3582b3ae94dc31c70a43872d255ff98cd99230da0ce86917582d893"}
Sep 16 19:13:09.625270 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.625247    2582 scope.go:117] "RemoveContainer" containerID="42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d"
Sep 16 19:13:09.633137 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.633119    2582 scope.go:117] "RemoveContainer" containerID="987a35905187188345af6fe3600bf49b075e503b74022a66c6258962664375b5"
Sep 16 19:13:09.641422 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.641402    2582 scope.go:117] "RemoveContainer" containerID="1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f"
Sep 16 19:13:09.641741 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:13:09.641723    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f\": container with ID starting with 1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f not found: ID does not exist" containerID="1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f"
Sep 16 19:13:09.641792 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.641748    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f"} err="failed to get container status \"1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f\": rpc error: code = NotFound desc = could not find container \"1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f\": container with ID starting with 1bac1c6bba939140305d226d44338211a9b747f77506340a22585e6cbd335a6f not found: ID does not exist"
Sep 16 19:13:09.641792 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.641765    2582 scope.go:117] "RemoveContainer" containerID="42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d"
Sep 16 19:13:09.642058 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:13:09.642036    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d\": container with ID starting with 42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d not found: ID does not exist" containerID="42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d"
Sep 16 19:13:09.642147 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.642067    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d"} err="failed to get container status \"42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d\": rpc error: code = NotFound desc = could not find container \"42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d\": container with ID starting with 42ebb31210b8f3ebc5c5ab746fa64dd6678c4f25e6d4474ffec22fa1bda1cb7d not found: ID does not exist"
Sep 16 19:13:09.642147 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.642088    2582 scope.go:117] "RemoveContainer" containerID="987a35905187188345af6fe3600bf49b075e503b74022a66c6258962664375b5"
Sep 16 19:13:09.642419 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:13:09.642397    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987a35905187188345af6fe3600bf49b075e503b74022a66c6258962664375b5\": container with ID starting with 987a35905187188345af6fe3600bf49b075e503b74022a66c6258962664375b5 not found: ID does not exist" containerID="987a35905187188345af6fe3600bf49b075e503b74022a66c6258962664375b5"
Sep 16 19:13:09.642475 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.642422    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987a35905187188345af6fe3600bf49b075e503b74022a66c6258962664375b5"} err="failed to get container status \"987a35905187188345af6fe3600bf49b075e503b74022a66c6258962664375b5\": rpc error: code = NotFound desc = could not find container \"987a35905187188345af6fe3600bf49b075e503b74022a66c6258962664375b5\": container with ID starting with 987a35905187188345af6fe3600bf49b075e503b74022a66c6258962664375b5 not found: ID does not exist"
Sep 16 19:13:09.700646 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.700626    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"]
Sep 16 19:13:09.707738 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:09.707721    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m"]
Sep 16 19:13:10.628848 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:10.628812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" event={"ID":"bc73837a-ce73-45f1-b2d8-fd3df2229464","Type":"ContainerStarted","Data":"febfe2d3bfff5b8b75bf8ba33e22a9e76d4a43b88974cbebc704ad2509da013c"}
Sep 16 19:13:10.629322 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:10.628882    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" event={"ID":"bc73837a-ce73-45f1-b2d8-fd3df2229464","Type":"ContainerStarted","Data":"fecb9d2bb8a4f91dd9bb5ed70ed3760c0eb275cb0924ace0b2d01c8df13fa885"}
Sep 16 19:13:10.629322 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:10.629210    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:10.629432 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:10.629383    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:10.630484 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:10.630449    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:13:10.673799 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:10.673750    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podStartSLOduration=2.118739476 podStartE2EDuration="5.673735564s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.554996088s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:10.668042912 +0000 UTC m=+1985.855418699" watchObservedRunningTime="2026-09-16 19:13:10.673735564 +0000 UTC m=+1985.861111351"
Sep 16 19:13:11.296698 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:11.296664    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" path="/var/lib/kubelet/pods/e25bb26a-18c2-4b40-9fff-2e8f21199c06/volumes"
Sep 16 19:13:11.634933 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:11.634834    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:13:16.641023 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:16.640996    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:13:16.641603 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:16.641573    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:13:26.641773 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:26.641734    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:13:36.642122 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:36.642080    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:13:46.641453 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:46.641410    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:13:56.641450 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:13:56.641409    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:14:06.641655 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:06.641616    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:14:16.642126 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:16.642083    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:14:26.642140 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:26.642047    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:14:35.234313 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:35.234272    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"]
Sep 16 19:14:35.234758 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:35.234513    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container" containerID="cri-o://fecb9d2bb8a4f91dd9bb5ed70ed3760c0eb275cb0924ace0b2d01c8df13fa885" gracePeriod=30
Sep 16 19:14:35.234758 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:35.234561    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kube-rbac-proxy" containerID="cri-o://febfe2d3bfff5b8b75bf8ba33e22a9e76d4a43b88974cbebc704ad2509da013c" gracePeriod=30
Sep 16 19:14:36.043204 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:36.043173    2582 generic.go:356] "Generic (PLEG): container finished" podID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerID="febfe2d3bfff5b8b75bf8ba33e22a9e76d4a43b88974cbebc704ad2509da013c" exitCode=2
Sep 16 19:14:36.043372 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:36.043210    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" event={"ID":"bc73837a-ce73-45f1-b2d8-fd3df2229464","Type":"ContainerDied","Data":"febfe2d3bfff5b8b75bf8ba33e22a9e76d4a43b88974cbebc704ad2509da013c"}
Sep 16 19:14:36.635825 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:36.635787    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 16 19:14:36.642242 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:36.642211    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:14:39.059260 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.059230    2582 generic.go:356] "Generic (PLEG): container finished" podID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerID="fecb9d2bb8a4f91dd9bb5ed70ed3760c0eb275cb0924ace0b2d01c8df13fa885" exitCode=0
Sep 16 19:14:39.059640 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.059293    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" event={"ID":"bc73837a-ce73-45f1-b2d8-fd3df2229464","Type":"ContainerDied","Data":"fecb9d2bb8a4f91dd9bb5ed70ed3760c0eb275cb0924ace0b2d01c8df13fa885"}
Sep 16 19:14:39.079340 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.079320    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:14:39.097895 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.097874    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc73837a-ce73-45f1-b2d8-fd3df2229464-proxy-tls\") pod \"bc73837a-ce73-45f1-b2d8-fd3df2229464\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") "
Sep 16 19:14:39.098009 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.097940    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc73837a-ce73-45f1-b2d8-fd3df2229464-kserve-provision-location\") pod \"bc73837a-ce73-45f1-b2d8-fd3df2229464\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") "
Sep 16 19:14:39.098009 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.097989    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc73837a-ce73-45f1-b2d8-fd3df2229464-kube-rbac-proxy-sar-config\") pod \"bc73837a-ce73-45f1-b2d8-fd3df2229464\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") "
Sep 16 19:14:39.098090 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.098016    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc73837a-ce73-45f1-b2d8-fd3df2229464-cabundle-cert\") pod \"bc73837a-ce73-45f1-b2d8-fd3df2229464\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") "
Sep 16 19:14:39.098090 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.098045    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhwn8\" (UniqueName: \"kubernetes.io/projected/bc73837a-ce73-45f1-b2d8-fd3df2229464-kube-api-access-rhwn8\") pod \"bc73837a-ce73-45f1-b2d8-fd3df2229464\" (UID: \"bc73837a-ce73-45f1-b2d8-fd3df2229464\") "
Sep 16 19:14:39.098296 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.098249    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc73837a-ce73-45f1-b2d8-fd3df2229464-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc73837a-ce73-45f1-b2d8-fd3df2229464" (UID: "bc73837a-ce73-45f1-b2d8-fd3df2229464"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:39.098434 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.098351    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc73837a-ce73-45f1-b2d8-fd3df2229464-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bc73837a-ce73-45f1-b2d8-fd3df2229464" (UID: "bc73837a-ce73-45f1-b2d8-fd3df2229464"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:39.098524 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.098500    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc73837a-ce73-45f1-b2d8-fd3df2229464-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:14:39.098617 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.098527    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc73837a-ce73-45f1-b2d8-fd3df2229464-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:14:39.098617 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.098578    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc73837a-ce73-45f1-b2d8-fd3df2229464-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bc73837a-ce73-45f1-b2d8-fd3df2229464" (UID: "bc73837a-ce73-45f1-b2d8-fd3df2229464"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:39.100013 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.099984    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc73837a-ce73-45f1-b2d8-fd3df2229464-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc73837a-ce73-45f1-b2d8-fd3df2229464" (UID: "bc73837a-ce73-45f1-b2d8-fd3df2229464"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:39.100280 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.100264    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc73837a-ce73-45f1-b2d8-fd3df2229464-kube-api-access-rhwn8" (OuterVolumeSpecName: "kube-api-access-rhwn8") pod "bc73837a-ce73-45f1-b2d8-fd3df2229464" (UID: "bc73837a-ce73-45f1-b2d8-fd3df2229464"). InnerVolumeSpecName "kube-api-access-rhwn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:39.199730 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.199707    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc73837a-ce73-45f1-b2d8-fd3df2229464-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:14:39.199730 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.199729    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc73837a-ce73-45f1-b2d8-fd3df2229464-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:14:39.199874 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:39.199743    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhwn8\" (UniqueName: \"kubernetes.io/projected/bc73837a-ce73-45f1-b2d8-fd3df2229464-kube-api-access-rhwn8\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:14:40.071295 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.071260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj" event={"ID":"bc73837a-ce73-45f1-b2d8-fd3df2229464","Type":"ContainerDied","Data":"f5c184bcc20c4bd406b5c3ff84c2612188d137cabac0ee49e143a252632a7435"}
Sep 16 19:14:40.071295 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.071297    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"
Sep 16 19:14:40.071825 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.071363    2582 scope.go:117] "RemoveContainer" containerID="febfe2d3bfff5b8b75bf8ba33e22a9e76d4a43b88974cbebc704ad2509da013c"
Sep 16 19:14:40.079715 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.079696    2582 scope.go:117] "RemoveContainer" containerID="fecb9d2bb8a4f91dd9bb5ed70ed3760c0eb275cb0924ace0b2d01c8df13fa885"
Sep 16 19:14:40.087369 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.087355    2582 scope.go:117] "RemoveContainer" containerID="9cc0877cc3582b3ae94dc31c70a43872d255ff98cd99230da0ce86917582d893"
Sep 16 19:14:40.100185 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.100165    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"]
Sep 16 19:14:40.103811 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.103792    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj"]
Sep 16 19:14:40.580511 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580482    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"]
Sep 16 19:14:40.580808 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580796    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="storage-initializer"
Sep 16 19:14:40.580854 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580810    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="storage-initializer"
Sep 16 19:14:40.580854 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580818    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container"
Sep 16 19:14:40.580854 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580824    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container"
Sep 16 19:14:40.580854 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580838    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="storage-initializer"
Sep 16 19:14:40.580854 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580844    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="storage-initializer"
Sep 16 19:14:40.580854 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580850    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container"
Sep 16 19:14:40.580854 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580855    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container"
Sep 16 19:14:40.581161 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580867    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kube-rbac-proxy"
Sep 16 19:14:40.581161 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580872    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kube-rbac-proxy"
Sep 16 19:14:40.581161 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580881    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kube-rbac-proxy"
Sep 16 19:14:40.581161 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580886    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kube-rbac-proxy"
Sep 16 19:14:40.581161 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580950    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kube-rbac-proxy"
Sep 16 19:14:40.581161 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580959    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kserve-container"
Sep 16 19:14:40.581161 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580964    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e25bb26a-18c2-4b40-9fff-2e8f21199c06" containerName="kserve-container"
Sep 16 19:14:40.581161 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.580970    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" containerName="kube-rbac-proxy"
Sep 16 19:14:40.585744 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.585725    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.598503 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.598484    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:14:40.598711 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.598698    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:14:40.598924 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.598890    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 19:14:40.599112 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.599099    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:14:40.599163 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.599146    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:14:40.599207 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.599146    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:14:40.599322 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.599305    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:14:40.607670 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.607646    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"]
Sep 16 19:14:40.610016 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.609995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.610104 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.610039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.610104 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.610063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvtf\" (UniqueName: \"kubernetes.io/projected/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kube-api-access-dnvtf\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.610180 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.610122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.610180 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.610157    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.710987 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.710967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.711092 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.711006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.711092 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.711056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.711092 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.711087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvtf\" (UniqueName: \"kubernetes.io/projected/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kube-api-access-dnvtf\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.711218 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.711152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.711218 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:14:40.711173    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:14:40.711311 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:14:40.711222    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-proxy-tls podName:b97499eb-ccf3-4c1d-80eb-94c80842dc0e nodeName:}" failed. No retries permitted until 2026-09-16 19:14:41.211208791 +0000 UTC m=+2076.398584581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-proxy-tls") pod "isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" (UID: "b97499eb-ccf3-4c1d-80eb-94c80842dc0e") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:14:40.711486 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.711466    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.711729 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.711711    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.711768 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.711755    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:40.726170 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:40.726148    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvtf\" (UniqueName: \"kubernetes.io/projected/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kube-api-access-dnvtf\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:41.214167 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:41.214134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:41.216820 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:41.216797    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:41.296886 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:41.296858    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc73837a-ce73-45f1-b2d8-fd3df2229464" path="/var/lib/kubelet/pods/bc73837a-ce73-45f1-b2d8-fd3df2229464/volumes"
Sep 16 19:14:41.495580 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:41.495513    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:41.781203 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:41.781179    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"]
Sep 16 19:14:41.783626 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:14:41.783597    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97499eb_ccf3_4c1d_80eb_94c80842dc0e.slice/crio-288525f8120521701da4b56110d80d9a9b2acc3bfa6249e982c70cc8892b3f6c WatchSource:0}: Error finding container 288525f8120521701da4b56110d80d9a9b2acc3bfa6249e982c70cc8892b3f6c: Status 404 returned error can't find the container with id 288525f8120521701da4b56110d80d9a9b2acc3bfa6249e982c70cc8892b3f6c
Sep 16 19:14:42.085040 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:42.084940    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" event={"ID":"b97499eb-ccf3-4c1d-80eb-94c80842dc0e","Type":"ContainerStarted","Data":"4ca842a35c54968de07a1a1aea68e13ed8f9dd1be3b1f61d6375702acd428842"}
Sep 16 19:14:42.085040 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:42.085010    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" event={"ID":"b97499eb-ccf3-4c1d-80eb-94c80842dc0e","Type":"ContainerStarted","Data":"288525f8120521701da4b56110d80d9a9b2acc3bfa6249e982c70cc8892b3f6c"}
Sep 16 19:14:44.781056 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.781029    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"]
Sep 16 19:14:44.784519 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.784502    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.787286 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.787259    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:14:44.787442 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.787354    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 16 19:14:44.802930 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.802892    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"]
Sep 16 19:14:44.841628 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.841596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.841628 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.841629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bzpx\" (UniqueName: \"kubernetes.io/projected/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kube-api-access-8bzpx\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.841820 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.841653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.841820 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.841668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.841820 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.841773    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.942479 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.942453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.942479 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.942481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bzpx\" (UniqueName: \"kubernetes.io/projected/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kube-api-access-8bzpx\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.942647 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.942502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.942647 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.942519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.942647 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.942557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.942819 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:14:44.942677    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 16 19:14:44.942819 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:14:44.942737    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-proxy-tls podName:d6530f39-7b4b-4d6a-8115-ef72eb5a6335 nodeName:}" failed. No retries permitted until 2026-09-16 19:14:45.44271824 +0000 UTC m=+2080.630094007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" (UID: "d6530f39-7b4b-4d6a-8115-ef72eb5a6335") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 16 19:14:44.942970 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.942933    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.943266 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.943245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.943354 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.943318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:44.958625 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:44.958602    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bzpx\" (UniqueName: \"kubernetes.io/projected/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kube-api-access-8bzpx\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:45.447790 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:45.447759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:45.450347 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:45.450327    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:45.701019 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:45.700894    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:45.992048 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:45.989445    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"]
Sep 16 19:14:46.106423 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:46.106396    2582 generic.go:356] "Generic (PLEG): container finished" podID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerID="4ca842a35c54968de07a1a1aea68e13ed8f9dd1be3b1f61d6375702acd428842" exitCode=0
Sep 16 19:14:46.106536 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:46.106471    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" event={"ID":"b97499eb-ccf3-4c1d-80eb-94c80842dc0e","Type":"ContainerDied","Data":"4ca842a35c54968de07a1a1aea68e13ed8f9dd1be3b1f61d6375702acd428842"}
Sep 16 19:14:46.108208 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:46.108185    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" event={"ID":"d6530f39-7b4b-4d6a-8115-ef72eb5a6335","Type":"ContainerStarted","Data":"9647239ab08f84557af0201122eac74a66cb083feb6e04dd7af5387c0be5f230"}
Sep 16 19:14:46.108312 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:46.108233    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" event={"ID":"d6530f39-7b4b-4d6a-8115-ef72eb5a6335","Type":"ContainerStarted","Data":"69637d4574eb4240edf32758c7fdaf72add6299b71de265c701dd82c98e42ea4"}
Sep 16 19:14:47.115245 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:47.115210    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" event={"ID":"b97499eb-ccf3-4c1d-80eb-94c80842dc0e","Type":"ContainerStarted","Data":"5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab"}
Sep 16 19:14:47.115658 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:47.115273    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" event={"ID":"b97499eb-ccf3-4c1d-80eb-94c80842dc0e","Type":"ContainerStarted","Data":"24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f"}
Sep 16 19:14:47.115658 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:47.115500    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:47.115658 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:47.115556    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:47.116855 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:47.116829    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:14:47.172752 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:47.172707    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podStartSLOduration=3.526661258 podStartE2EDuration="7.17269389s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.646032632s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:47.157244785 +0000 UTC m=+2082.344620577" watchObservedRunningTime="2026-09-16 19:14:47.17269389 +0000 UTC m=+2082.360069676"
Sep 16 19:14:48.122110 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:48.122065    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:14:51.138338 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:51.138303    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerID="9647239ab08f84557af0201122eac74a66cb083feb6e04dd7af5387c0be5f230" exitCode=0
Sep 16 19:14:51.138704 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:51.138378    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" event={"ID":"d6530f39-7b4b-4d6a-8115-ef72eb5a6335","Type":"ContainerDied","Data":"9647239ab08f84557af0201122eac74a66cb083feb6e04dd7af5387c0be5f230"}
Sep 16 19:14:51.139530 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:51.139513    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:14:53.127363 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:53.127329    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:14:53.128116 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:53.128075    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:14:55.162756 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:55.162679    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" event={"ID":"d6530f39-7b4b-4d6a-8115-ef72eb5a6335","Type":"ContainerStarted","Data":"247fd4d29d447432232f6a7fbd30e2dd3a70233d95f4707d5fd33ce5635bd39c"}
Sep 16 19:14:55.162756 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:55.162740    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" event={"ID":"d6530f39-7b4b-4d6a-8115-ef72eb5a6335","Type":"ContainerStarted","Data":"2c07a61cd0dc18beb5fd8a6d4921f911806db87fbe25f37f32f0e38a90cbb086"}
Sep 16 19:14:55.163173 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:55.163035    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:55.194384 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:55.194340    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" podStartSLOduration=3.413023292 podStartE2EDuration="11.194325737s" totalImagesPullingTime="3.631457881s" totalInitContainerRuntime="4.149844564s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:55.192443954 +0000 UTC m=+2090.379819752" watchObservedRunningTime="2026-09-16 19:14:55.194325737 +0000 UTC m=+2090.381701570"
Sep 16 19:14:56.167712 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:56.167680    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:14:56.168987 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:56.168960    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 16 19:14:57.172356 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:14:57.172313    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 16 19:15:02.177117 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:02.177090    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:15:02.177599 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:02.177536    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 16 19:15:03.128045 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:03.128008    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:15:05.389189 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:05.389162    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:15:05.395669 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:05.395648    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:15:12.178290 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:12.178261    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:15:13.128023 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:13.127985    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:15:23.128879 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:23.128837    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:15:26.288589 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:26.288555    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"]
Sep 16 19:15:26.288993 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:26.288810    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kserve-container" containerID="cri-o://2c07a61cd0dc18beb5fd8a6d4921f911806db87fbe25f37f32f0e38a90cbb086" gracePeriod=30
Sep 16 19:15:26.288993 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:26.288900    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kube-rbac-proxy" containerID="cri-o://247fd4d29d447432232f6a7fbd30e2dd3a70233d95f4707d5fd33ce5635bd39c" gracePeriod=30
Sep 16 19:15:27.173589 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:27.173538    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 16 19:15:27.328047 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:27.328011    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerID="247fd4d29d447432232f6a7fbd30e2dd3a70233d95f4707d5fd33ce5635bd39c" exitCode=2
Sep 16 19:15:27.328430 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:27.328079    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" event={"ID":"d6530f39-7b4b-4d6a-8115-ef72eb5a6335","Type":"ContainerDied","Data":"247fd4d29d447432232f6a7fbd30e2dd3a70233d95f4707d5fd33ce5635bd39c"}
Sep 16 19:15:32.172835 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:32.172784    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 16 19:15:33.128230 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:33.128187    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:15:37.172772 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:37.172730    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 16 19:15:37.173196 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:37.172873    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:15:42.173608 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:42.173563    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 16 19:15:43.127964 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:43.127927    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:15:47.172881 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:47.172837    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 16 19:15:52.173220 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:52.173178    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 16 19:15:53.128523 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:53.128480    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:15:56.479948 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:56.479856    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerID="2c07a61cd0dc18beb5fd8a6d4921f911806db87fbe25f37f32f0e38a90cbb086" exitCode=137
Sep 16 19:15:56.480306 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:56.479936    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" event={"ID":"d6530f39-7b4b-4d6a-8115-ef72eb5a6335","Type":"ContainerDied","Data":"2c07a61cd0dc18beb5fd8a6d4921f911806db87fbe25f37f32f0e38a90cbb086"}
Sep 16 19:15:56.936530 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:56.936504    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:15:57.075611 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.075513    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kube-rbac-proxy-sar-config\") pod \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") "
Sep 16 19:15:57.075611 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.075564    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bzpx\" (UniqueName: \"kubernetes.io/projected/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kube-api-access-8bzpx\") pod \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") "
Sep 16 19:15:57.075611 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.075603    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kserve-provision-location\") pod \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") "
Sep 16 19:15:57.075936 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.075623    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-proxy-tls\") pod \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") "
Sep 16 19:15:57.075936 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.075649    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-cabundle-cert\") pod \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\" (UID: \"d6530f39-7b4b-4d6a-8115-ef72eb5a6335\") "
Sep 16 19:15:57.076077 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.076047    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6530f39-7b4b-4d6a-8115-ef72eb5a6335" (UID: "d6530f39-7b4b-4d6a-8115-ef72eb5a6335"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:57.076126 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.076098    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6530f39-7b4b-4d6a-8115-ef72eb5a6335" (UID: "d6530f39-7b4b-4d6a-8115-ef72eb5a6335"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:57.077950 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.077924    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kube-api-access-8bzpx" (OuterVolumeSpecName: "kube-api-access-8bzpx") pod "d6530f39-7b4b-4d6a-8115-ef72eb5a6335" (UID: "d6530f39-7b4b-4d6a-8115-ef72eb5a6335"). InnerVolumeSpecName "kube-api-access-8bzpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:57.078071 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.078017    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6530f39-7b4b-4d6a-8115-ef72eb5a6335" (UID: "d6530f39-7b4b-4d6a-8115-ef72eb5a6335"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:57.087219 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.087190    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6530f39-7b4b-4d6a-8115-ef72eb5a6335" (UID: "d6530f39-7b4b-4d6a-8115-ef72eb5a6335"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:57.176491 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.176454    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bzpx\" (UniqueName: \"kubernetes.io/projected/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kube-api-access-8bzpx\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:15:57.176491 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.176486    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:15:57.176491 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.176496    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:15:57.176736 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.176507    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:15:57.176736 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.176515    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6530f39-7b4b-4d6a-8115-ef72eb5a6335-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:15:57.493009 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.492968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt" event={"ID":"d6530f39-7b4b-4d6a-8115-ef72eb5a6335","Type":"ContainerDied","Data":"69637d4574eb4240edf32758c7fdaf72add6299b71de265c701dd82c98e42ea4"}
Sep 16 19:15:57.493428 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.493072    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"
Sep 16 19:15:57.493428 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.493086    2582 scope.go:117] "RemoveContainer" containerID="247fd4d29d447432232f6a7fbd30e2dd3a70233d95f4707d5fd33ce5635bd39c"
Sep 16 19:15:57.501801 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.501782    2582 scope.go:117] "RemoveContainer" containerID="2c07a61cd0dc18beb5fd8a6d4921f911806db87fbe25f37f32f0e38a90cbb086"
Sep 16 19:15:57.509778 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.509755    2582 scope.go:117] "RemoveContainer" containerID="9647239ab08f84557af0201122eac74a66cb083feb6e04dd7af5387c0be5f230"
Sep 16 19:15:57.524482 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.524459    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"]
Sep 16 19:15:57.531123 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:57.531101    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt"]
Sep 16 19:15:59.296230 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:15:59.296194    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" path="/var/lib/kubelet/pods/d6530f39-7b4b-4d6a-8115-ef72eb5a6335/volumes"
Sep 16 19:16:03.128850 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:16:03.128814    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:16:09.450694 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:16:09.450658    2582 scope.go:117] "RemoveContainer" containerID="f07b0800476460b5a2430c181369f34df6c2d44effc94e0a87999e51e1199d22"
Sep 16 19:16:09.459684 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:16:09.459665    2582 scope.go:117] "RemoveContainer" containerID="5703065ed9f063274d0475262fb37b035adb0e16fc4824a711980ba39e8e6705"
Sep 16 19:16:09.467463 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:16:09.467447    2582 scope.go:117] "RemoveContainer" containerID="76cc06ddc59d58fcef79e96deebcc537fe98f308b8b7cea9e4774a916c5a87e9"
Sep 16 19:18:50.776596 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:50.776560    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"]
Sep 16 19:18:50.777132 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:50.776813    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container" containerID="cri-o://24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f" gracePeriod=30
Sep 16 19:18:50.777132 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:50.776843    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kube-rbac-proxy" containerID="cri-o://5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab" gracePeriod=30
Sep 16 19:18:51.273715 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:51.273680    2582 generic.go:356] "Generic (PLEG): container finished" podID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerID="5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab" exitCode=2
Sep 16 19:18:51.273889 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:51.273738    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" event={"ID":"b97499eb-ccf3-4c1d-80eb-94c80842dc0e","Type":"ContainerDied","Data":"5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab"}
Sep 16 19:18:53.123314 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:53.123272    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 16 19:18:53.128647 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:53.128621    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:18:55.821549 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:55.821522    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:18:55.907393 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:55.907306    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kube-rbac-proxy-sar-config\") pod \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") "
Sep 16 19:18:55.907393 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:55.907347    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kserve-provision-location\") pod \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") "
Sep 16 19:18:55.907589 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:55.907406    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-cabundle-cert\") pod \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") "
Sep 16 19:18:55.907589 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:55.907450    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnvtf\" (UniqueName: \"kubernetes.io/projected/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kube-api-access-dnvtf\") pod \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") "
Sep 16 19:18:55.907589 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:55.907492    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-proxy-tls\") pod \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\" (UID: \"b97499eb-ccf3-4c1d-80eb-94c80842dc0e\") "
Sep 16 19:18:55.907761 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:55.907732    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b97499eb-ccf3-4c1d-80eb-94c80842dc0e" (UID: "b97499eb-ccf3-4c1d-80eb-94c80842dc0e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:55.907809 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:55.907753    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b97499eb-ccf3-4c1d-80eb-94c80842dc0e" (UID: "b97499eb-ccf3-4c1d-80eb-94c80842dc0e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:55.907809 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:55.907771    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b97499eb-ccf3-4c1d-80eb-94c80842dc0e" (UID: "b97499eb-ccf3-4c1d-80eb-94c80842dc0e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:55.909935 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:55.909886    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b97499eb-ccf3-4c1d-80eb-94c80842dc0e" (UID: "b97499eb-ccf3-4c1d-80eb-94c80842dc0e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:55.909935 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:55.909893    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kube-api-access-dnvtf" (OuterVolumeSpecName: "kube-api-access-dnvtf") pod "b97499eb-ccf3-4c1d-80eb-94c80842dc0e" (UID: "b97499eb-ccf3-4c1d-80eb-94c80842dc0e"). InnerVolumeSpecName "kube-api-access-dnvtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:56.000479 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000438    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"]
Sep 16 19:18:56.000788 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000774    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container"
Sep 16 19:18:56.000832 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000790    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container"
Sep 16 19:18:56.000832 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000811    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="storage-initializer"
Sep 16 19:18:56.000832 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000817    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="storage-initializer"
Sep 16 19:18:56.000832 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000825    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="storage-initializer"
Sep 16 19:18:56.000832 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000831    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="storage-initializer"
Sep 16 19:18:56.001010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000838    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kube-rbac-proxy"
Sep 16 19:18:56.001010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000844    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kube-rbac-proxy"
Sep 16 19:18:56.001010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000854    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kserve-container"
Sep 16 19:18:56.001010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000860    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kserve-container"
Sep 16 19:18:56.001010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000870    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kube-rbac-proxy"
Sep 16 19:18:56.001010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000875    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kube-rbac-proxy"
Sep 16 19:18:56.001010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000947    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kserve-container"
Sep 16 19:18:56.001010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000956    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerName="kube-rbac-proxy"
Sep 16 19:18:56.001010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000961    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kserve-container"
Sep 16 19:18:56.001010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.000969    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6530f39-7b4b-4d6a-8115-ef72eb5a6335" containerName="kube-rbac-proxy"
Sep 16 19:18:56.004385 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.004363    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.008115 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.008083    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 19:18:56.008450 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.008423    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.008537 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.008454    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvtf\" (UniqueName: \"kubernetes.io/projected/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kube-api-access-dnvtf\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.008537 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.008471    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.008537 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.008485    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.008537 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.008498    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b97499eb-ccf3-4c1d-80eb-94c80842dc0e-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.012209 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.012188    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 19:18:56.020404 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.020377    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"]
Sep 16 19:18:56.109204 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.109158    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f1020f-6856-4f56-a437-8af2f13806cb-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.109418 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.109244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r22xs\" (UniqueName: \"kubernetes.io/projected/98f1020f-6856-4f56-a437-8af2f13806cb-kube-api-access-r22xs\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.109418 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.109272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f1020f-6856-4f56-a437-8af2f13806cb-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.109418 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.109338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f1020f-6856-4f56-a437-8af2f13806cb-proxy-tls\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.109418 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.109367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f1020f-6856-4f56-a437-8af2f13806cb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.210697 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.210658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r22xs\" (UniqueName: \"kubernetes.io/projected/98f1020f-6856-4f56-a437-8af2f13806cb-kube-api-access-r22xs\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.210697 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.210699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f1020f-6856-4f56-a437-8af2f13806cb-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.211015 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.210757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f1020f-6856-4f56-a437-8af2f13806cb-proxy-tls\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.211015 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.210787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f1020f-6856-4f56-a437-8af2f13806cb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.211015 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.210852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f1020f-6856-4f56-a437-8af2f13806cb-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.211438 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.211414    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f1020f-6856-4f56-a437-8af2f13806cb-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.211583 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.211559    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f1020f-6856-4f56-a437-8af2f13806cb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.211656 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.211631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f1020f-6856-4f56-a437-8af2f13806cb-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.213537 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.213520    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f1020f-6856-4f56-a437-8af2f13806cb-proxy-tls\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.220886 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.220859    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r22xs\" (UniqueName: \"kubernetes.io/projected/98f1020f-6856-4f56-a437-8af2f13806cb-kube-api-access-r22xs\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-fbb6q\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.303210 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.303163    2582 generic.go:356] "Generic (PLEG): container finished" podID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" containerID="24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f" exitCode=0
Sep 16 19:18:56.303420 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.303296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" event={"ID":"b97499eb-ccf3-4c1d-80eb-94c80842dc0e","Type":"ContainerDied","Data":"24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f"}
Sep 16 19:18:56.303420 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.303351    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl" event={"ID":"b97499eb-ccf3-4c1d-80eb-94c80842dc0e","Type":"ContainerDied","Data":"288525f8120521701da4b56110d80d9a9b2acc3bfa6249e982c70cc8892b3f6c"}
Sep 16 19:18:56.303420 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.303393    2582 scope.go:117] "RemoveContainer" containerID="5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab"
Sep 16 19:18:56.303587 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.303570    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"
Sep 16 19:18:56.313438 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.313365    2582 scope.go:117] "RemoveContainer" containerID="24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f"
Sep 16 19:18:56.315702 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.315681    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:18:56.323549 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.323528    2582 scope.go:117] "RemoveContainer" containerID="4ca842a35c54968de07a1a1aea68e13ed8f9dd1be3b1f61d6375702acd428842"
Sep 16 19:18:56.334093 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.334031    2582 scope.go:117] "RemoveContainer" containerID="5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab"
Sep 16 19:18:56.334518 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:18:56.334487    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab\": container with ID starting with 5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab not found: ID does not exist" containerID="5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab"
Sep 16 19:18:56.334658 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.334531    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab"} err="failed to get container status \"5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab\": rpc error: code = NotFound desc = could not find container \"5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab\": container with ID starting with 5ff294a6c0898b442f99758e621a83aafda7ada5fa298037375f61f69835d0ab not found: ID does not exist"
Sep 16 19:18:56.334658 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.334562    2582 scope.go:117] "RemoveContainer" containerID="24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f"
Sep 16 19:18:56.334988 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:18:56.334960    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f\": container with ID starting with 24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f not found: ID does not exist" containerID="24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f"
Sep 16 19:18:56.335085 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.334996    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f"} err="failed to get container status \"24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f\": rpc error: code = NotFound desc = could not find container \"24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f\": container with ID starting with 24eccc956d8871a8ab04761c0757f449b2fbcbd145b24beb34c0001674193b0f not found: ID does not exist"
Sep 16 19:18:56.335085 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.335019    2582 scope.go:117] "RemoveContainer" containerID="4ca842a35c54968de07a1a1aea68e13ed8f9dd1be3b1f61d6375702acd428842"
Sep 16 19:18:56.335465 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:18:56.335342    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca842a35c54968de07a1a1aea68e13ed8f9dd1be3b1f61d6375702acd428842\": container with ID starting with 4ca842a35c54968de07a1a1aea68e13ed8f9dd1be3b1f61d6375702acd428842 not found: ID does not exist" containerID="4ca842a35c54968de07a1a1aea68e13ed8f9dd1be3b1f61d6375702acd428842"
Sep 16 19:18:56.335465 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.335381    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca842a35c54968de07a1a1aea68e13ed8f9dd1be3b1f61d6375702acd428842"} err="failed to get container status \"4ca842a35c54968de07a1a1aea68e13ed8f9dd1be3b1f61d6375702acd428842\": rpc error: code = NotFound desc = could not find container \"4ca842a35c54968de07a1a1aea68e13ed8f9dd1be3b1f61d6375702acd428842\": container with ID starting with 4ca842a35c54968de07a1a1aea68e13ed8f9dd1be3b1f61d6375702acd428842 not found: ID does not exist"
Sep 16 19:18:56.337300 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.337278    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"]
Sep 16 19:18:56.349055 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.349023    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl"]
Sep 16 19:18:56.611863 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:56.611837    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"]
Sep 16 19:18:56.614040 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:18:56.614009    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f1020f_6856_4f56_a437_8af2f13806cb.slice/crio-0f4b112e4fb6845d918f88e5160a9bb5ecb7bdb148ca9db4391c6080c356ab71 WatchSource:0}: Error finding container 0f4b112e4fb6845d918f88e5160a9bb5ecb7bdb148ca9db4391c6080c356ab71: Status 404 returned error can't find the container with id 0f4b112e4fb6845d918f88e5160a9bb5ecb7bdb148ca9db4391c6080c356ab71
Sep 16 19:18:57.297701 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:57.297664    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b97499eb-ccf3-4c1d-80eb-94c80842dc0e" path="/var/lib/kubelet/pods/b97499eb-ccf3-4c1d-80eb-94c80842dc0e/volumes"
Sep 16 19:18:57.316968 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:57.316923    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" event={"ID":"98f1020f-6856-4f56-a437-8af2f13806cb","Type":"ContainerStarted","Data":"81ef18ad1984966624d3cb1a1316c3d98e21d73a4114ed76f97c43db8d771bed"}
Sep 16 19:18:57.317168 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:18:57.317029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" event={"ID":"98f1020f-6856-4f56-a437-8af2f13806cb","Type":"ContainerStarted","Data":"0f4b112e4fb6845d918f88e5160a9bb5ecb7bdb148ca9db4391c6080c356ab71"}
Sep 16 19:19:01.340384 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:01.340350    2582 generic.go:356] "Generic (PLEG): container finished" podID="98f1020f-6856-4f56-a437-8af2f13806cb" containerID="81ef18ad1984966624d3cb1a1316c3d98e21d73a4114ed76f97c43db8d771bed" exitCode=0
Sep 16 19:19:01.340754 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:01.340426    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" event={"ID":"98f1020f-6856-4f56-a437-8af2f13806cb","Type":"ContainerDied","Data":"81ef18ad1984966624d3cb1a1316c3d98e21d73a4114ed76f97c43db8d771bed"}
Sep 16 19:19:02.347608 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:02.347573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" event={"ID":"98f1020f-6856-4f56-a437-8af2f13806cb","Type":"ContainerStarted","Data":"b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80"}
Sep 16 19:19:02.348079 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:02.347638    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" event={"ID":"98f1020f-6856-4f56-a437-8af2f13806cb","Type":"ContainerStarted","Data":"d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b"}
Sep 16 19:19:02.348079 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:02.347864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:19:02.378375 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:02.378234    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podStartSLOduration=3.494916405 podStartE2EDuration="7.378212225s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.88329582s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:02.37754978 +0000 UTC m=+2337.564925566" watchObservedRunningTime="2026-09-16 19:19:02.378212225 +0000 UTC m=+2337.565588014"
Sep 16 19:19:03.352947 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:03.352895    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:19:03.354566 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:03.354537    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:19:04.357860 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:04.357819    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:19:09.362490 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:09.362461    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:19:09.362986 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:09.362961    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:19:19.363791 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:19.363750    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:19:29.362868 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:29.362826    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:19:34.678280 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.678245    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"]
Sep 16 19:19:34.681969 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.681953    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.686680 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.686658    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 16 19:19:34.686680 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.686672    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:19:34.700185 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.700166    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"]
Sep 16 19:19:34.829825 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.829803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6r4s\" (UniqueName: \"kubernetes.io/projected/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kube-api-access-l6r4s\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.830021 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.829846    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40b45bdb-05d6-43e1-8777-b5d2e89c2531-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.830021 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.829890    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.830021 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.829933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.830021 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.829969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40b45bdb-05d6-43e1-8777-b5d2e89c2531-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.930509 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.930435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.930611 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.930555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40b45bdb-05d6-43e1-8777-b5d2e89c2531-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.930799 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.930619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6r4s\" (UniqueName: \"kubernetes.io/projected/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kube-api-access-l6r4s\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.930799 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.930657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40b45bdb-05d6-43e1-8777-b5d2e89c2531-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.930799 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:19:34.930658    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:19:34.930799 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.930691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.930799 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:19:34.930736    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b45bdb-05d6-43e1-8777-b5d2e89c2531-proxy-tls podName:40b45bdb-05d6-43e1-8777-b5d2e89c2531 nodeName:}" failed. No retries permitted until 2026-09-16 19:19:35.430717354 +0000 UTC m=+2370.618093132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/40b45bdb-05d6-43e1-8777-b5d2e89c2531-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" (UID: "40b45bdb-05d6-43e1-8777-b5d2e89c2531") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:19:34.931125 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.931038    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.931234 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.931203    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.931279 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.931271    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40b45bdb-05d6-43e1-8777-b5d2e89c2531-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:34.943455 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:34.943430    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6r4s\" (UniqueName: \"kubernetes.io/projected/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kube-api-access-l6r4s\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:35.435424 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:35.435400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40b45bdb-05d6-43e1-8777-b5d2e89c2531-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:35.437898 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:35.437876    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40b45bdb-05d6-43e1-8777-b5d2e89c2531-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:35.592380 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:35.592351    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:35.882333 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:35.880496    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"]
Sep 16 19:19:35.885634 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:19:35.885602    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b45bdb_05d6_43e1_8777_b5d2e89c2531.slice/crio-adb7e7bf08a784fd82f5873ef0a9f557a07e3c76e8eee08a0a5854bea5830fe8 WatchSource:0}: Error finding container adb7e7bf08a784fd82f5873ef0a9f557a07e3c76e8eee08a0a5854bea5830fe8: Status 404 returned error can't find the container with id adb7e7bf08a784fd82f5873ef0a9f557a07e3c76e8eee08a0a5854bea5830fe8
Sep 16 19:19:36.495687 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:36.495654    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" event={"ID":"40b45bdb-05d6-43e1-8777-b5d2e89c2531","Type":"ContainerStarted","Data":"bcbcbc911f053e713cc604a5288279ccdb0d423009ca4fbd4ad0c797d6bc8356"}
Sep 16 19:19:36.495920 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:36.495711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" event={"ID":"40b45bdb-05d6-43e1-8777-b5d2e89c2531","Type":"ContainerStarted","Data":"adb7e7bf08a784fd82f5873ef0a9f557a07e3c76e8eee08a0a5854bea5830fe8"}
Sep 16 19:19:39.362918 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:39.362878    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:19:40.515759 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:40.515730    2582 generic.go:356] "Generic (PLEG): container finished" podID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerID="bcbcbc911f053e713cc604a5288279ccdb0d423009ca4fbd4ad0c797d6bc8356" exitCode=0
Sep 16 19:19:40.516167 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:40.515774    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" event={"ID":"40b45bdb-05d6-43e1-8777-b5d2e89c2531","Type":"ContainerDied","Data":"bcbcbc911f053e713cc604a5288279ccdb0d423009ca4fbd4ad0c797d6bc8356"}
Sep 16 19:19:41.523757 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:41.523711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" event={"ID":"40b45bdb-05d6-43e1-8777-b5d2e89c2531","Type":"ContainerStarted","Data":"a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608"}
Sep 16 19:19:41.524162 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:41.523805    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" event={"ID":"40b45bdb-05d6-43e1-8777-b5d2e89c2531","Type":"ContainerStarted","Data":"1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9"}
Sep 16 19:19:41.524162 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:41.524058    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:41.554351 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:41.554289    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" podStartSLOduration=3.71734537 podStartE2EDuration="7.554274067s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.836928697s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:41.54993525 +0000 UTC m=+2376.737311030" watchObservedRunningTime="2026-09-16 19:19:41.554274067 +0000 UTC m=+2376.741649904"
Sep 16 19:19:42.528635 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:42.528606    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:48.538957 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:48.538923    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:19:49.363725 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:49.363682    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:19:59.362854 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:19:59.362820    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:20:05.424434 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:05.424404    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:20:05.432033 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:05.432008    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:20:09.363308 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:09.363266    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:20:17.296303 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:17.296275    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:20:18.542601 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:18.542568    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:20:24.759094 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:24.759062    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"]
Sep 16 19:20:24.759481 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:24.759312    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerName="kserve-container" containerID="cri-o://1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9" gracePeriod=30
Sep 16 19:20:24.759481 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:24.759339    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerName="kube-rbac-proxy" containerID="cri-o://a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608" gracePeriod=30
Sep 16 19:20:25.730508 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:25.730475    2582 generic.go:356] "Generic (PLEG): container finished" podID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerID="a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608" exitCode=2
Sep 16 19:20:25.730676 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:25.730547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" event={"ID":"40b45bdb-05d6-43e1-8777-b5d2e89c2531","Type":"ContainerDied","Data":"a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608"}
Sep 16 19:20:26.077162 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:26.077068    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"]
Sep 16 19:20:26.077595 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:26.077356    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container" containerID="cri-o://d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b" gracePeriod=30
Sep 16 19:20:26.077595 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:26.077403    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kube-rbac-proxy" containerID="cri-o://b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80" gracePeriod=30
Sep 16 19:20:26.738067 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:26.737994    2582 generic.go:356] "Generic (PLEG): container finished" podID="98f1020f-6856-4f56-a437-8af2f13806cb" containerID="b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80" exitCode=2
Sep 16 19:20:26.738214 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:26.738069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" event={"ID":"98f1020f-6856-4f56-a437-8af2f13806cb","Type":"ContainerDied","Data":"b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80"}
Sep 16 19:20:27.292606 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:27.292564    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:20:28.534262 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:28.534226    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 16 19:20:28.539676 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:28.539654    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.48:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 16 19:20:29.358786 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:29.358749    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 16 19:20:30.207893 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.207864    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"]
Sep 16 19:20:30.211471 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.211451    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.213992 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.213952    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:20:30.214089 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.214070    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 16 19:20:30.230031 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.230008    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"]
Sep 16 19:20:30.365019 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.364993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb709fb6-0792-4ce3-a13b-69c678732297-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.365129 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.365024    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb709fb6-0792-4ce3-a13b-69c678732297-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.365225 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.365194    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb709fb6-0792-4ce3-a13b-69c678732297-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.365525 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.365503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h52ts\" (UniqueName: \"kubernetes.io/projected/eb709fb6-0792-4ce3-a13b-69c678732297-kube-api-access-h52ts\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.365585 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.365554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb709fb6-0792-4ce3-a13b-69c678732297-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.403966 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.403939    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:20:30.466873 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.466844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb709fb6-0792-4ce3-a13b-69c678732297-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.467027 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.466893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h52ts\" (UniqueName: \"kubernetes.io/projected/eb709fb6-0792-4ce3-a13b-69c678732297-kube-api-access-h52ts\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.467027 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.466936    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb709fb6-0792-4ce3-a13b-69c678732297-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.467149 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.467100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb709fb6-0792-4ce3-a13b-69c678732297-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.467204 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.467144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb709fb6-0792-4ce3-a13b-69c678732297-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.467262 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.467247    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb709fb6-0792-4ce3-a13b-69c678732297-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.467649 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.467621    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb709fb6-0792-4ce3-a13b-69c678732297-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.467749 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.467652    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb709fb6-0792-4ce3-a13b-69c678732297-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.469612 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.469595    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb709fb6-0792-4ce3-a13b-69c678732297-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.479281 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.479258    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h52ts\" (UniqueName: \"kubernetes.io/projected/eb709fb6-0792-4ce3-a13b-69c678732297-kube-api-access-h52ts\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.491946 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.491929    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:20:30.527374 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.527352    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:30.568304 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568276    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kserve-provision-location\") pod \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") "
Sep 16 19:20:30.568464 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568353    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f1020f-6856-4f56-a437-8af2f13806cb-kserve-provision-location\") pod \"98f1020f-6856-4f56-a437-8af2f13806cb\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") "
Sep 16 19:20:30.568464 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568382    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f1020f-6856-4f56-a437-8af2f13806cb-cabundle-cert\") pod \"98f1020f-6856-4f56-a437-8af2f13806cb\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") "
Sep 16 19:20:30.568464 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568408    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kube-rbac-proxy-sar-config\") pod \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") "
Sep 16 19:20:30.568464 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568457    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40b45bdb-05d6-43e1-8777-b5d2e89c2531-cabundle-cert\") pod \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") "
Sep 16 19:20:30.568698 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568483    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r22xs\" (UniqueName: \"kubernetes.io/projected/98f1020f-6856-4f56-a437-8af2f13806cb-kube-api-access-r22xs\") pod \"98f1020f-6856-4f56-a437-8af2f13806cb\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") "
Sep 16 19:20:30.568698 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568516    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40b45bdb-05d6-43e1-8777-b5d2e89c2531-proxy-tls\") pod \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") "
Sep 16 19:20:30.568698 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568544    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f1020f-6856-4f56-a437-8af2f13806cb-kube-rbac-proxy-sar-config\") pod \"98f1020f-6856-4f56-a437-8af2f13806cb\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") "
Sep 16 19:20:30.568698 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568569    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f1020f-6856-4f56-a437-8af2f13806cb-proxy-tls\") pod \"98f1020f-6856-4f56-a437-8af2f13806cb\" (UID: \"98f1020f-6856-4f56-a437-8af2f13806cb\") "
Sep 16 19:20:30.568698 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568596    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6r4s\" (UniqueName: \"kubernetes.io/projected/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kube-api-access-l6r4s\") pod \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\" (UID: \"40b45bdb-05d6-43e1-8777-b5d2e89c2531\") "
Sep 16 19:20:30.568698 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568662    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40b45bdb-05d6-43e1-8777-b5d2e89c2531" (UID: "40b45bdb-05d6-43e1-8777-b5d2e89c2531"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:30.569022 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568720    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f1020f-6856-4f56-a437-8af2f13806cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98f1020f-6856-4f56-a437-8af2f13806cb" (UID: "98f1020f-6856-4f56-a437-8af2f13806cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:30.569022 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568810    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40b45bdb-05d6-43e1-8777-b5d2e89c2531" (UID: "40b45bdb-05d6-43e1-8777-b5d2e89c2531"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:30.569022 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.568817    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b45bdb-05d6-43e1-8777-b5d2e89c2531-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40b45bdb-05d6-43e1-8777-b5d2e89c2531" (UID: "40b45bdb-05d6-43e1-8777-b5d2e89c2531"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:30.569195 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.569121    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f1020f-6856-4f56-a437-8af2f13806cb-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:20:30.569195 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.569142    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:20:30.569195 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.569156    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40b45bdb-05d6-43e1-8777-b5d2e89c2531-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:20:30.569195 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.569170    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:20:30.569570 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.569204    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f1020f-6856-4f56-a437-8af2f13806cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98f1020f-6856-4f56-a437-8af2f13806cb" (UID: "98f1020f-6856-4f56-a437-8af2f13806cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:30.569570 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.569262    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f1020f-6856-4f56-a437-8af2f13806cb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "98f1020f-6856-4f56-a437-8af2f13806cb" (UID: "98f1020f-6856-4f56-a437-8af2f13806cb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:30.570962 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.570877    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f1020f-6856-4f56-a437-8af2f13806cb-kube-api-access-r22xs" (OuterVolumeSpecName: "kube-api-access-r22xs") pod "98f1020f-6856-4f56-a437-8af2f13806cb" (UID: "98f1020f-6856-4f56-a437-8af2f13806cb"). InnerVolumeSpecName "kube-api-access-r22xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:30.570962 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.570880    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kube-api-access-l6r4s" (OuterVolumeSpecName: "kube-api-access-l6r4s") pod "40b45bdb-05d6-43e1-8777-b5d2e89c2531" (UID: "40b45bdb-05d6-43e1-8777-b5d2e89c2531"). InnerVolumeSpecName "kube-api-access-l6r4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:30.571483 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.571456    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f1020f-6856-4f56-a437-8af2f13806cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98f1020f-6856-4f56-a437-8af2f13806cb" (UID: "98f1020f-6856-4f56-a437-8af2f13806cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:30.571483 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.571463    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b45bdb-05d6-43e1-8777-b5d2e89c2531-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40b45bdb-05d6-43e1-8777-b5d2e89c2531" (UID: "40b45bdb-05d6-43e1-8777-b5d2e89c2531"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:30.670218 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.670162    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f1020f-6856-4f56-a437-8af2f13806cb-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:20:30.670218 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.670186    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r22xs\" (UniqueName: \"kubernetes.io/projected/98f1020f-6856-4f56-a437-8af2f13806cb-kube-api-access-r22xs\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:20:30.670218 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.670197    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40b45bdb-05d6-43e1-8777-b5d2e89c2531-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:20:30.670218 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.670208    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f1020f-6856-4f56-a437-8af2f13806cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:20:30.670218 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.670216    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f1020f-6856-4f56-a437-8af2f13806cb-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:20:30.670454 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.670224    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6r4s\" (UniqueName: \"kubernetes.io/projected/40b45bdb-05d6-43e1-8777-b5d2e89c2531-kube-api-access-l6r4s\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:20:30.759328 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.759300    2582 generic.go:356] "Generic (PLEG): container finished" podID="98f1020f-6856-4f56-a437-8af2f13806cb" containerID="d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b" exitCode=0
Sep 16 19:20:30.759472 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.759369    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"
Sep 16 19:20:30.759472 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.759374    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" event={"ID":"98f1020f-6856-4f56-a437-8af2f13806cb","Type":"ContainerDied","Data":"d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b"}
Sep 16 19:20:30.759472 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.759434    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q" event={"ID":"98f1020f-6856-4f56-a437-8af2f13806cb","Type":"ContainerDied","Data":"0f4b112e4fb6845d918f88e5160a9bb5ecb7bdb148ca9db4391c6080c356ab71"}
Sep 16 19:20:30.759472 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.759459    2582 scope.go:117] "RemoveContainer" containerID="b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80"
Sep 16 19:20:30.761879 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.761857    2582 generic.go:356] "Generic (PLEG): container finished" podID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerID="1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9" exitCode=0
Sep 16 19:20:30.762010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.761891    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" event={"ID":"40b45bdb-05d6-43e1-8777-b5d2e89c2531","Type":"ContainerDied","Data":"1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9"}
Sep 16 19:20:30.762010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.761900    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"
Sep 16 19:20:30.762010 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.761965    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9" event={"ID":"40b45bdb-05d6-43e1-8777-b5d2e89c2531","Type":"ContainerDied","Data":"adb7e7bf08a784fd82f5873ef0a9f557a07e3c76e8eee08a0a5854bea5830fe8"}
Sep 16 19:20:30.769296 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.769275    2582 scope.go:117] "RemoveContainer" containerID="d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b"
Sep 16 19:20:30.776961 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.776946    2582 scope.go:117] "RemoveContainer" containerID="81ef18ad1984966624d3cb1a1316c3d98e21d73a4114ed76f97c43db8d771bed"
Sep 16 19:20:30.784606 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.784590    2582 scope.go:117] "RemoveContainer" containerID="b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80"
Sep 16 19:20:30.784866 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:20:30.784847    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80\": container with ID starting with b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80 not found: ID does not exist" containerID="b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80"
Sep 16 19:20:30.784972 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.784872    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80"} err="failed to get container status \"b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80\": rpc error: code = NotFound desc = could not find container \"b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80\": container with ID starting with b417b549b512b43f55e941675276f70b7a3acaeb92b2cbea154bf862f4d73a80 not found: ID does not exist"
Sep 16 19:20:30.784972 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.784887    2582 scope.go:117] "RemoveContainer" containerID="d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b"
Sep 16 19:20:30.785167 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:20:30.785153    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b\": container with ID starting with d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b not found: ID does not exist" containerID="d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b"
Sep 16 19:20:30.785209 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.785171    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b"} err="failed to get container status \"d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b\": rpc error: code = NotFound desc = could not find container \"d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b\": container with ID starting with d095d2c3158efd951a1e9db0d318ddae564f4a888bf640fd10487e21d967835b not found: ID does not exist"
Sep 16 19:20:30.785209 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.785183    2582 scope.go:117] "RemoveContainer" containerID="81ef18ad1984966624d3cb1a1316c3d98e21d73a4114ed76f97c43db8d771bed"
Sep 16 19:20:30.785418 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:20:30.785403    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ef18ad1984966624d3cb1a1316c3d98e21d73a4114ed76f97c43db8d771bed\": container with ID starting with 81ef18ad1984966624d3cb1a1316c3d98e21d73a4114ed76f97c43db8d771bed not found: ID does not exist" containerID="81ef18ad1984966624d3cb1a1316c3d98e21d73a4114ed76f97c43db8d771bed"
Sep 16 19:20:30.785470 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.785420    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ef18ad1984966624d3cb1a1316c3d98e21d73a4114ed76f97c43db8d771bed"} err="failed to get container status \"81ef18ad1984966624d3cb1a1316c3d98e21d73a4114ed76f97c43db8d771bed\": rpc error: code = NotFound desc = could not find container \"81ef18ad1984966624d3cb1a1316c3d98e21d73a4114ed76f97c43db8d771bed\": container with ID starting with 81ef18ad1984966624d3cb1a1316c3d98e21d73a4114ed76f97c43db8d771bed not found: ID does not exist"
Sep 16 19:20:30.785470 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.785434    2582 scope.go:117] "RemoveContainer" containerID="a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608"
Sep 16 19:20:30.793070 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.793049    2582 scope.go:117] "RemoveContainer" containerID="1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9"
Sep 16 19:20:30.800219 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.800202    2582 scope.go:117] "RemoveContainer" containerID="bcbcbc911f053e713cc604a5288279ccdb0d423009ca4fbd4ad0c797d6bc8356"
Sep 16 19:20:30.807235 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.807221    2582 scope.go:117] "RemoveContainer" containerID="a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608"
Sep 16 19:20:30.807495 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:20:30.807479    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608\": container with ID starting with a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608 not found: ID does not exist" containerID="a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608"
Sep 16 19:20:30.807541 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.807499    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608"} err="failed to get container status \"a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608\": rpc error: code = NotFound desc = could not find container \"a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608\": container with ID starting with a2c0497b38c5041a27da5b0baa7047afbdd3f8ce5eda330da7313026c4e9f608 not found: ID does not exist"
Sep 16 19:20:30.807541 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.807513    2582 scope.go:117] "RemoveContainer" containerID="1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9"
Sep 16 19:20:30.807752 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:20:30.807737    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9\": container with ID starting with 1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9 not found: ID does not exist" containerID="1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9"
Sep 16 19:20:30.807802 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.807753    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9"} err="failed to get container status \"1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9\": rpc error: code = NotFound desc = could not find container \"1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9\": container with ID starting with 1d9c52ccb9bb44ac7afb5369d4b4be98d7f337f29e8f70a9cbea966f170ab5e9 not found: ID does not exist"
Sep 16 19:20:30.807802 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.807765    2582 scope.go:117] "RemoveContainer" containerID="bcbcbc911f053e713cc604a5288279ccdb0d423009ca4fbd4ad0c797d6bc8356"
Sep 16 19:20:30.807997 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:20:30.807980    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcbcbc911f053e713cc604a5288279ccdb0d423009ca4fbd4ad0c797d6bc8356\": container with ID starting with bcbcbc911f053e713cc604a5288279ccdb0d423009ca4fbd4ad0c797d6bc8356 not found: ID does not exist" containerID="bcbcbc911f053e713cc604a5288279ccdb0d423009ca4fbd4ad0c797d6bc8356"
Sep 16 19:20:30.808036 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.808000    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcbcbc911f053e713cc604a5288279ccdb0d423009ca4fbd4ad0c797d6bc8356"} err="failed to get container status \"bcbcbc911f053e713cc604a5288279ccdb0d423009ca4fbd4ad0c797d6bc8356\": rpc error: code = NotFound desc = could not find container \"bcbcbc911f053e713cc604a5288279ccdb0d423009ca4fbd4ad0c797d6bc8356\": container with ID starting with bcbcbc911f053e713cc604a5288279ccdb0d423009ca4fbd4ad0c797d6bc8356 not found: ID does not exist"
Sep 16 19:20:30.818306 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.818284    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"]
Sep 16 19:20:30.824824 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.824803    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"]
Sep 16 19:20:30.828773 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:20:30.828749    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb709fb6_0792_4ce3_a13b_69c678732297.slice/crio-fb7feb750ccd3c982f40e728cd9ce11085b7b08413521670aa2572f1682546c1 WatchSource:0}: Error finding container fb7feb750ccd3c982f40e728cd9ce11085b7b08413521670aa2572f1682546c1: Status 404 returned error can't find the container with id fb7feb750ccd3c982f40e728cd9ce11085b7b08413521670aa2572f1682546c1
Sep 16 19:20:30.829857 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.829837    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9"]
Sep 16 19:20:30.859089 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.859060    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"]
Sep 16 19:20:30.864671 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:30.864652    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q"]
Sep 16 19:20:31.297771 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:31.297730    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" path="/var/lib/kubelet/pods/40b45bdb-05d6-43e1-8777-b5d2e89c2531/volumes"
Sep 16 19:20:31.298391 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:31.298371    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" path="/var/lib/kubelet/pods/98f1020f-6856-4f56-a437-8af2f13806cb/volumes"
Sep 16 19:20:31.770467 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:31.770439    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" event={"ID":"eb709fb6-0792-4ce3-a13b-69c678732297","Type":"ContainerStarted","Data":"81890e0189162a1e4543621b3ac11daa8b3d3d3ba754585462f5640d0977c1ab"}
Sep 16 19:20:31.770638 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:31.770489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" event={"ID":"eb709fb6-0792-4ce3-a13b-69c678732297","Type":"ContainerStarted","Data":"fb7feb750ccd3c982f40e728cd9ce11085b7b08413521670aa2572f1682546c1"}
Sep 16 19:20:35.790862 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:35.790830    2582 generic.go:356] "Generic (PLEG): container finished" podID="eb709fb6-0792-4ce3-a13b-69c678732297" containerID="81890e0189162a1e4543621b3ac11daa8b3d3d3ba754585462f5640d0977c1ab" exitCode=0
Sep 16 19:20:35.791281 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:35.790895    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" event={"ID":"eb709fb6-0792-4ce3-a13b-69c678732297","Type":"ContainerDied","Data":"81890e0189162a1e4543621b3ac11daa8b3d3d3ba754585462f5640d0977c1ab"}
Sep 16 19:20:36.797535 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:36.797502    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" event={"ID":"eb709fb6-0792-4ce3-a13b-69c678732297","Type":"ContainerStarted","Data":"79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d"}
Sep 16 19:20:36.797896 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:36.797562    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" event={"ID":"eb709fb6-0792-4ce3-a13b-69c678732297","Type":"ContainerStarted","Data":"8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173"}
Sep 16 19:20:36.797896 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:36.797755    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:36.797991 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:36.797932    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:20:36.831602 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:36.831556    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" podStartSLOduration=2.840548995 podStartE2EDuration="6.831543714s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.990994719s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:36.828241893 +0000 UTC m=+2432.015617680" watchObservedRunningTime="2026-09-16 19:20:36.831543714 +0000 UTC m=+2432.018919503"
Sep 16 19:20:42.808879 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:20:42.808851    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:21:12.813370 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:12.813335    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:21:20.179179 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:20.179142    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"]
Sep 16 19:21:20.179691 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:20.179358    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" containerName="kserve-container" containerID="cri-o://8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173" gracePeriod=30
Sep 16 19:21:20.179691 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:20.179428    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" containerName="kube-rbac-proxy" containerID="cri-o://79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d" gracePeriod=30
Sep 16 19:21:21.006139 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:21.006102    2582 generic.go:356] "Generic (PLEG): container finished" podID="eb709fb6-0792-4ce3-a13b-69c678732297" containerID="79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d" exitCode=2
Sep 16 19:21:21.006139 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:21.006142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" event={"ID":"eb709fb6-0792-4ce3-a13b-69c678732297","Type":"ContainerDied","Data":"79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d"}
Sep 16 19:21:22.803475 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:22.803434    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 16 19:21:22.809951 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:22.809894    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.49:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 16 19:21:25.219724 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.219703    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:21:25.278592 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.278524    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb709fb6-0792-4ce3-a13b-69c678732297-cabundle-cert\") pod \"eb709fb6-0792-4ce3-a13b-69c678732297\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") "
Sep 16 19:21:25.278723 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.278636    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb709fb6-0792-4ce3-a13b-69c678732297-kserve-provision-location\") pod \"eb709fb6-0792-4ce3-a13b-69c678732297\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") "
Sep 16 19:21:25.278723 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.278679    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb709fb6-0792-4ce3-a13b-69c678732297-kube-rbac-proxy-sar-config\") pod \"eb709fb6-0792-4ce3-a13b-69c678732297\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") "
Sep 16 19:21:25.278723 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.278709    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb709fb6-0792-4ce3-a13b-69c678732297-proxy-tls\") pod \"eb709fb6-0792-4ce3-a13b-69c678732297\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") "
Sep 16 19:21:25.278881 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.278742    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h52ts\" (UniqueName: \"kubernetes.io/projected/eb709fb6-0792-4ce3-a13b-69c678732297-kube-api-access-h52ts\") pod \"eb709fb6-0792-4ce3-a13b-69c678732297\" (UID: \"eb709fb6-0792-4ce3-a13b-69c678732297\") "
Sep 16 19:21:25.279073 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.279048    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb709fb6-0792-4ce3-a13b-69c678732297-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb709fb6-0792-4ce3-a13b-69c678732297" (UID: "eb709fb6-0792-4ce3-a13b-69c678732297"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:25.279129 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.279056    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb709fb6-0792-4ce3-a13b-69c678732297-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb709fb6-0792-4ce3-a13b-69c678732297" (UID: "eb709fb6-0792-4ce3-a13b-69c678732297"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:25.279129 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.279119    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb709fb6-0792-4ce3-a13b-69c678732297-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb709fb6-0792-4ce3-a13b-69c678732297" (UID: "eb709fb6-0792-4ce3-a13b-69c678732297"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:25.281068 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.281042    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb709fb6-0792-4ce3-a13b-69c678732297-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb709fb6-0792-4ce3-a13b-69c678732297" (UID: "eb709fb6-0792-4ce3-a13b-69c678732297"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:25.281178 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.281088    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb709fb6-0792-4ce3-a13b-69c678732297-kube-api-access-h52ts" (OuterVolumeSpecName: "kube-api-access-h52ts") pod "eb709fb6-0792-4ce3-a13b-69c678732297" (UID: "eb709fb6-0792-4ce3-a13b-69c678732297"). InnerVolumeSpecName "kube-api-access-h52ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:25.379567 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.379544    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb709fb6-0792-4ce3-a13b-69c678732297-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:21:25.379567 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.379564    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb709fb6-0792-4ce3-a13b-69c678732297-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:21:25.379706 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.379575    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb709fb6-0792-4ce3-a13b-69c678732297-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:21:25.379706 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.379586    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h52ts\" (UniqueName: \"kubernetes.io/projected/eb709fb6-0792-4ce3-a13b-69c678732297-kube-api-access-h52ts\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:21:25.379706 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:25.379594    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb709fb6-0792-4ce3-a13b-69c678732297-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.029835 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.029808    2582 generic.go:356] "Generic (PLEG): container finished" podID="eb709fb6-0792-4ce3-a13b-69c678732297" containerID="8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173" exitCode=0
Sep 16 19:21:26.030057 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.029883    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"
Sep 16 19:21:26.030057 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.029900    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" event={"ID":"eb709fb6-0792-4ce3-a13b-69c678732297","Type":"ContainerDied","Data":"8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173"}
Sep 16 19:21:26.030057 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.029991    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv" event={"ID":"eb709fb6-0792-4ce3-a13b-69c678732297","Type":"ContainerDied","Data":"fb7feb750ccd3c982f40e728cd9ce11085b7b08413521670aa2572f1682546c1"}
Sep 16 19:21:26.030057 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.030019    2582 scope.go:117] "RemoveContainer" containerID="79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d"
Sep 16 19:21:26.038954 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.038934    2582 scope.go:117] "RemoveContainer" containerID="8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173"
Sep 16 19:21:26.047104 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.047088    2582 scope.go:117] "RemoveContainer" containerID="81890e0189162a1e4543621b3ac11daa8b3d3d3ba754585462f5640d0977c1ab"
Sep 16 19:21:26.055058 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.055035    2582 scope.go:117] "RemoveContainer" containerID="79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d"
Sep 16 19:21:26.055346 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:21:26.055326    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d\": container with ID starting with 79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d not found: ID does not exist" containerID="79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d"
Sep 16 19:21:26.055427 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.055353    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d"} err="failed to get container status \"79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d\": rpc error: code = NotFound desc = could not find container \"79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d\": container with ID starting with 79a9ab70f7d1b5f87adb2a55348171be110260650ddad8b12611564e43f47c0d not found: ID does not exist"
Sep 16 19:21:26.055427 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.055383    2582 scope.go:117] "RemoveContainer" containerID="8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173"
Sep 16 19:21:26.055657 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:21:26.055638    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173\": container with ID starting with 8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173 not found: ID does not exist" containerID="8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173"
Sep 16 19:21:26.055703 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.055666    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173"} err="failed to get container status \"8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173\": rpc error: code = NotFound desc = could not find container \"8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173\": container with ID starting with 8bc382faf283ed5d1a0c52b629151b0e047179a6f049d6a5ff56538b27bb5173 not found: ID does not exist"
Sep 16 19:21:26.055703 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.055682    2582 scope.go:117] "RemoveContainer" containerID="81890e0189162a1e4543621b3ac11daa8b3d3d3ba754585462f5640d0977c1ab"
Sep 16 19:21:26.055939 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:21:26.055899    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81890e0189162a1e4543621b3ac11daa8b3d3d3ba754585462f5640d0977c1ab\": container with ID starting with 81890e0189162a1e4543621b3ac11daa8b3d3d3ba754585462f5640d0977c1ab not found: ID does not exist" containerID="81890e0189162a1e4543621b3ac11daa8b3d3d3ba754585462f5640d0977c1ab"
Sep 16 19:21:26.055990 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.055947    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81890e0189162a1e4543621b3ac11daa8b3d3d3ba754585462f5640d0977c1ab"} err="failed to get container status \"81890e0189162a1e4543621b3ac11daa8b3d3d3ba754585462f5640d0977c1ab\": rpc error: code = NotFound desc = could not find container \"81890e0189162a1e4543621b3ac11daa8b3d3d3ba754585462f5640d0977c1ab\": container with ID starting with 81890e0189162a1e4543621b3ac11daa8b3d3d3ba754585462f5640d0977c1ab not found: ID does not exist"
Sep 16 19:21:26.077974 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.077950    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"]
Sep 16 19:21:26.088974 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:26.088955    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv"]
Sep 16 19:21:27.299607 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:21:27.299575    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" path="/var/lib/kubelet/pods/eb709fb6-0792-4ce3-a13b-69c678732297/volumes"
Sep 16 19:22:50.732323 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732291    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"]
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732615    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" containerName="kube-rbac-proxy"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732626    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" containerName="kube-rbac-proxy"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732639    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="storage-initializer"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732645    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="storage-initializer"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732657    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732663    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732671    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kube-rbac-proxy"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732677    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kube-rbac-proxy"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732683    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerName="storage-initializer"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732688    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerName="storage-initializer"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732697    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" containerName="kserve-container"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732702    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" containerName="kserve-container"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732708    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerName="kube-rbac-proxy"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732714    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerName="kube-rbac-proxy"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732720    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" containerName="storage-initializer"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732725    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" containerName="storage-initializer"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732740    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerName="kserve-container"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732745    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerName="kserve-container"
Sep 16 19:22:50.732800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732810    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kserve-container"
Sep 16 19:22:50.733413 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732817    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" containerName="kserve-container"
Sep 16 19:22:50.733413 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732825    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f1020f-6856-4f56-a437-8af2f13806cb" containerName="kube-rbac-proxy"
Sep 16 19:22:50.733413 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732830    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerName="kserve-container"
Sep 16 19:22:50.733413 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732836    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb709fb6-0792-4ce3-a13b-69c678732297" containerName="kube-rbac-proxy"
Sep 16 19:22:50.733413 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.732842    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="40b45bdb-05d6-43e1-8777-b5d2e89c2531" containerName="kube-rbac-proxy"
Sep 16 19:22:50.736056 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.736039    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.740841 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.740818    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 16 19:22:50.741833 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.741816    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:22:50.741958 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.741917    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:22:50.742035 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.741957    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:22:50.742094 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.741988    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:22:50.742221 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.742186    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:22:50.742994 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.742978    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:22:50.765130 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.765109    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"]
Sep 16 19:22:50.814873 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.814848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80349036-559d-40fb-8ce5-29d1d80b4d7a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.814980 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.814890    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md5v6\" (UniqueName: \"kubernetes.io/projected/80349036-559d-40fb-8ce5-29d1d80b4d7a-kube-api-access-md5v6\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.814980 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.814930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80349036-559d-40fb-8ce5-29d1d80b4d7a-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.815064 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.815003    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80349036-559d-40fb-8ce5-29d1d80b4d7a-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.815107 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.815058    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80349036-559d-40fb-8ce5-29d1d80b4d7a-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.915724 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.915703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80349036-559d-40fb-8ce5-29d1d80b4d7a-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.915825 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.915738    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80349036-559d-40fb-8ce5-29d1d80b4d7a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.915825 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.915769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md5v6\" (UniqueName: \"kubernetes.io/projected/80349036-559d-40fb-8ce5-29d1d80b4d7a-kube-api-access-md5v6\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.915825 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.915793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80349036-559d-40fb-8ce5-29d1d80b4d7a-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.916004 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.915839    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80349036-559d-40fb-8ce5-29d1d80b4d7a-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.916004 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:22:50.915977    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 16 19:22:50.916086 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:22:50.916055    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80349036-559d-40fb-8ce5-29d1d80b4d7a-proxy-tls podName:80349036-559d-40fb-8ce5-29d1d80b4d7a nodeName:}" failed. No retries permitted until 2026-09-16 19:22:51.416036887 +0000 UTC m=+2566.603412654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/80349036-559d-40fb-8ce5-29d1d80b4d7a-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" (UID: "80349036-559d-40fb-8ce5-29d1d80b4d7a") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 16 19:22:50.916387 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.916364    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80349036-559d-40fb-8ce5-29d1d80b4d7a-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.916491 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.916473    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80349036-559d-40fb-8ce5-29d1d80b4d7a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.916532 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.916494    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80349036-559d-40fb-8ce5-29d1d80b4d7a-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:50.934277 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:50.934257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md5v6\" (UniqueName: \"kubernetes.io/projected/80349036-559d-40fb-8ce5-29d1d80b4d7a-kube-api-access-md5v6\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:51.419899 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:51.419871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80349036-559d-40fb-8ce5-29d1d80b4d7a-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:51.422406 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:51.422384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80349036-559d-40fb-8ce5-29d1d80b4d7a-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:51.646147 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:51.646124    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:51.931835 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:51.931643    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"]
Sep 16 19:22:51.934279 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:22:51.934245    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80349036_559d_40fb_8ce5_29d1d80b4d7a.slice/crio-350b5b2f292df6c6365ff3d27d431b3e66e8c485e0ab49c05019ac8b7ed61a1d WatchSource:0}: Error finding container 350b5b2f292df6c6365ff3d27d431b3e66e8c485e0ab49c05019ac8b7ed61a1d: Status 404 returned error can't find the container with id 350b5b2f292df6c6365ff3d27d431b3e66e8c485e0ab49c05019ac8b7ed61a1d
Sep 16 19:22:52.415250 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:52.415166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" event={"ID":"80349036-559d-40fb-8ce5-29d1d80b4d7a","Type":"ContainerStarted","Data":"59eb8924629597dfe0017423643434e1c52e925c6f8a482e1a3ed2e6285489d0"}
Sep 16 19:22:52.415250 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:52.415226    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" event={"ID":"80349036-559d-40fb-8ce5-29d1d80b4d7a","Type":"ContainerStarted","Data":"350b5b2f292df6c6365ff3d27d431b3e66e8c485e0ab49c05019ac8b7ed61a1d"}
Sep 16 19:22:56.433857 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:56.433826    2582 generic.go:356] "Generic (PLEG): container finished" podID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerID="59eb8924629597dfe0017423643434e1c52e925c6f8a482e1a3ed2e6285489d0" exitCode=0
Sep 16 19:22:56.434230 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:56.433862    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" event={"ID":"80349036-559d-40fb-8ce5-29d1d80b4d7a","Type":"ContainerDied","Data":"59eb8924629597dfe0017423643434e1c52e925c6f8a482e1a3ed2e6285489d0"}
Sep 16 19:22:57.440167 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:57.440134    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" event={"ID":"80349036-559d-40fb-8ce5-29d1d80b4d7a","Type":"ContainerStarted","Data":"86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae"}
Sep 16 19:22:57.440632 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:57.440202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" event={"ID":"80349036-559d-40fb-8ce5-29d1d80b4d7a","Type":"ContainerStarted","Data":"fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663"}
Sep 16 19:22:57.440632 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:57.440402    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:57.440632 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:57.440448    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:22:57.478043 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:22:57.477999    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" podStartSLOduration=3.500004521 podStartE2EDuration="7.477984922s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.977980401s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:57.477827936 +0000 UTC m=+2572.665203724" watchObservedRunningTime="2026-09-16 19:22:57.477984922 +0000 UTC m=+2572.665360708"
Sep 16 19:23:03.450142 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:03.450109    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:23:33.519824 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:33.519743    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:23:43.452679 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:43.452649    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:23:48.101803 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.101767    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"]
Sep 16 19:23:48.105524 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.105507    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.111220 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.111197    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 16 19:23:48.111549 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.111528    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:23:48.123846 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.123824    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"]
Sep 16 19:23:48.127156 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.127135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xnkf\" (UniqueName: \"kubernetes.io/projected/bef4c0c0-5485-4e0b-953a-763e10070e1d-kube-api-access-7xnkf\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.127250 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.127165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef4c0c0-5485-4e0b-953a-763e10070e1d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.127250 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.127185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef4c0c0-5485-4e0b-953a-763e10070e1d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.127323 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.127265    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef4c0c0-5485-4e0b-953a-763e10070e1d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.127323 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.127305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef4c0c0-5485-4e0b-953a-763e10070e1d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.228610 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.228584    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef4c0c0-5485-4e0b-953a-763e10070e1d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.228709 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.228614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef4c0c0-5485-4e0b-953a-763e10070e1d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.228709 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.228644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef4c0c0-5485-4e0b-953a-763e10070e1d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.228709 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.228663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef4c0c0-5485-4e0b-953a-763e10070e1d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.228845 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.228723    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xnkf\" (UniqueName: \"kubernetes.io/projected/bef4c0c0-5485-4e0b-953a-763e10070e1d-kube-api-access-7xnkf\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.229121 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.229100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef4c0c0-5485-4e0b-953a-763e10070e1d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.229279 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.229260    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef4c0c0-5485-4e0b-953a-763e10070e1d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.229346 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.229324    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef4c0c0-5485-4e0b-953a-763e10070e1d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.231327 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.231306    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef4c0c0-5485-4e0b-953a-763e10070e1d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.242063 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.242041    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xnkf\" (UniqueName: \"kubernetes.io/projected/bef4c0c0-5485-4e0b-953a-763e10070e1d-kube-api-access-7xnkf\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.415650 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.415595    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:48.702512 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:48.702456    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"]
Sep 16 19:23:48.704804 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:23:48.704769    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef4c0c0_5485_4e0b_953a_763e10070e1d.slice/crio-09d3e0d5302151ff2f786f0902aa9073f2638ca741c27115cbf5c0d90e0efc21 WatchSource:0}: Error finding container 09d3e0d5302151ff2f786f0902aa9073f2638ca741c27115cbf5c0d90e0efc21: Status 404 returned error can't find the container with id 09d3e0d5302151ff2f786f0902aa9073f2638ca741c27115cbf5c0d90e0efc21
Sep 16 19:23:49.666260 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:49.666224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln" event={"ID":"bef4c0c0-5485-4e0b-953a-763e10070e1d","Type":"ContainerStarted","Data":"f4398b8de996c7174988c85ee658812e81710673eebb232d22d85c0510035eab"}
Sep 16 19:23:49.666643 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:49.666285    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln" event={"ID":"bef4c0c0-5485-4e0b-953a-763e10070e1d","Type":"ContainerStarted","Data":"09d3e0d5302151ff2f786f0902aa9073f2638ca741c27115cbf5c0d90e0efc21"}
Sep 16 19:23:50.787640 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:50.787602    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"]
Sep 16 19:23:50.788032 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:50.787829    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerName="kserve-container" containerID="cri-o://fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663" gracePeriod=30
Sep 16 19:23:50.788032 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:50.787860    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerName="kube-rbac-proxy" containerID="cri-o://86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae" gracePeriod=30
Sep 16 19:23:51.677121 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:51.677084    2582 generic.go:356] "Generic (PLEG): container finished" podID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerID="86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae" exitCode=2
Sep 16 19:23:51.677289 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:51.677153    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" event={"ID":"80349036-559d-40fb-8ce5-29d1d80b4d7a","Type":"ContainerDied","Data":"86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae"}
Sep 16 19:23:52.683539 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:52.683513    2582 generic.go:356] "Generic (PLEG): container finished" podID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerID="f4398b8de996c7174988c85ee658812e81710673eebb232d22d85c0510035eab" exitCode=0
Sep 16 19:23:52.683872 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:52.683547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln" event={"ID":"bef4c0c0-5485-4e0b-953a-763e10070e1d","Type":"ContainerDied","Data":"f4398b8de996c7174988c85ee658812e81710673eebb232d22d85c0510035eab"}
Sep 16 19:23:53.444996 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:53.444939    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 16 19:23:53.691584 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:53.691542    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln" event={"ID":"bef4c0c0-5485-4e0b-953a-763e10070e1d","Type":"ContainerStarted","Data":"8ddf5ad314026e44099f9a83fe7fb27a691b9df3c34f6461cc60d1f609ed42d6"}
Sep 16 19:23:53.692074 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:53.691624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln" event={"ID":"bef4c0c0-5485-4e0b-953a-763e10070e1d","Type":"ContainerStarted","Data":"88820eb44c5517c30c4f0942daa37d46c7fe457e955084411f934cb3150aa964"}
Sep 16 19:23:53.692074 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:53.691921    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:53.692196 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:53.692088    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:23:53.731951 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:53.731831    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln" podStartSLOduration=1.9772019699999999 podStartE2EDuration="5.731816999s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.754615029s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:53.731332762 +0000 UTC m=+2628.918708583" watchObservedRunningTime="2026-09-16 19:23:53.731816999 +0000 UTC m=+2628.919192785"
Sep 16 19:23:57.927937 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:57.927887    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:23:58.012934 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.012846    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80349036-559d-40fb-8ce5-29d1d80b4d7a-kserve-provision-location\") pod \"80349036-559d-40fb-8ce5-29d1d80b4d7a\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") "
Sep 16 19:23:58.012934 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.012918    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80349036-559d-40fb-8ce5-29d1d80b4d7a-proxy-tls\") pod \"80349036-559d-40fb-8ce5-29d1d80b4d7a\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") "
Sep 16 19:23:58.013141 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.012974    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80349036-559d-40fb-8ce5-29d1d80b4d7a-cabundle-cert\") pod \"80349036-559d-40fb-8ce5-29d1d80b4d7a\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") "
Sep 16 19:23:58.013141 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.013084    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80349036-559d-40fb-8ce5-29d1d80b4d7a-kube-rbac-proxy-sar-config\") pod \"80349036-559d-40fb-8ce5-29d1d80b4d7a\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") "
Sep 16 19:23:58.013229 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.013145    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md5v6\" (UniqueName: \"kubernetes.io/projected/80349036-559d-40fb-8ce5-29d1d80b4d7a-kube-api-access-md5v6\") pod \"80349036-559d-40fb-8ce5-29d1d80b4d7a\" (UID: \"80349036-559d-40fb-8ce5-29d1d80b4d7a\") "
Sep 16 19:23:58.013229 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.013210    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80349036-559d-40fb-8ce5-29d1d80b4d7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "80349036-559d-40fb-8ce5-29d1d80b4d7a" (UID: "80349036-559d-40fb-8ce5-29d1d80b4d7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:58.013328 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.013310    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80349036-559d-40fb-8ce5-29d1d80b4d7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "80349036-559d-40fb-8ce5-29d1d80b4d7a" (UID: "80349036-559d-40fb-8ce5-29d1d80b4d7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:58.013454 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.013437    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80349036-559d-40fb-8ce5-29d1d80b4d7a-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.013514 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.013462    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80349036-559d-40fb-8ce5-29d1d80b4d7a-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.013573 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.013552    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80349036-559d-40fb-8ce5-29d1d80b4d7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "80349036-559d-40fb-8ce5-29d1d80b4d7a" (UID: "80349036-559d-40fb-8ce5-29d1d80b4d7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:58.015261 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.015233    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80349036-559d-40fb-8ce5-29d1d80b4d7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80349036-559d-40fb-8ce5-29d1d80b4d7a" (UID: "80349036-559d-40fb-8ce5-29d1d80b4d7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:58.015462 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.015441    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80349036-559d-40fb-8ce5-29d1d80b4d7a-kube-api-access-md5v6" (OuterVolumeSpecName: "kube-api-access-md5v6") pod "80349036-559d-40fb-8ce5-29d1d80b4d7a" (UID: "80349036-559d-40fb-8ce5-29d1d80b4d7a"). InnerVolumeSpecName "kube-api-access-md5v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:58.114445 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.114418    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80349036-559d-40fb-8ce5-29d1d80b4d7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.114445 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.114441    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md5v6\" (UniqueName: \"kubernetes.io/projected/80349036-559d-40fb-8ce5-29d1d80b4d7a-kube-api-access-md5v6\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.114445 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.114453    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80349036-559d-40fb-8ce5-29d1d80b4d7a-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.719044 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.719007    2582 generic.go:356] "Generic (PLEG): container finished" podID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerID="fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663" exitCode=0
Sep 16 19:23:58.719221 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.719081    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"
Sep 16 19:23:58.719221 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.719094    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" event={"ID":"80349036-559d-40fb-8ce5-29d1d80b4d7a","Type":"ContainerDied","Data":"fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663"}
Sep 16 19:23:58.719221 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.719167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k" event={"ID":"80349036-559d-40fb-8ce5-29d1d80b4d7a","Type":"ContainerDied","Data":"350b5b2f292df6c6365ff3d27d431b3e66e8c485e0ab49c05019ac8b7ed61a1d"}
Sep 16 19:23:58.719221 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.719195    2582 scope.go:117] "RemoveContainer" containerID="86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae"
Sep 16 19:23:58.729756 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.729736    2582 scope.go:117] "RemoveContainer" containerID="fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663"
Sep 16 19:23:58.738040 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.738023    2582 scope.go:117] "RemoveContainer" containerID="59eb8924629597dfe0017423643434e1c52e925c6f8a482e1a3ed2e6285489d0"
Sep 16 19:23:58.746315 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.746293    2582 scope.go:117] "RemoveContainer" containerID="86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae"
Sep 16 19:23:58.746897 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:23:58.746739    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae\": container with ID starting with 86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae not found: ID does not exist" containerID="86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae"
Sep 16 19:23:58.746897 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.746805    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae"} err="failed to get container status \"86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae\": rpc error: code = NotFound desc = could not find container \"86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae\": container with ID starting with 86f68439ad51d9ffc6b467006c51e559e04b033d16d043b5aed785ed37d3b3ae not found: ID does not exist"
Sep 16 19:23:58.746897 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.746827    2582 scope.go:117] "RemoveContainer" containerID="fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663"
Sep 16 19:23:58.747533 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:23:58.747388    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663\": container with ID starting with fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663 not found: ID does not exist" containerID="fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663"
Sep 16 19:23:58.747533 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.747431    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663"} err="failed to get container status \"fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663\": rpc error: code = NotFound desc = could not find container \"fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663\": container with ID starting with fa92310ae4c6f51da9ac22a884082d9409f7307864fe14678b24276faee1d663 not found: ID does not exist"
Sep 16 19:23:58.747533 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.747453    2582 scope.go:117] "RemoveContainer" containerID="59eb8924629597dfe0017423643434e1c52e925c6f8a482e1a3ed2e6285489d0"
Sep 16 19:23:58.747785 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:23:58.747767    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59eb8924629597dfe0017423643434e1c52e925c6f8a482e1a3ed2e6285489d0\": container with ID starting with 59eb8924629597dfe0017423643434e1c52e925c6f8a482e1a3ed2e6285489d0 not found: ID does not exist" containerID="59eb8924629597dfe0017423643434e1c52e925c6f8a482e1a3ed2e6285489d0"
Sep 16 19:23:58.747828 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.747790    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59eb8924629597dfe0017423643434e1c52e925c6f8a482e1a3ed2e6285489d0"} err="failed to get container status \"59eb8924629597dfe0017423643434e1c52e925c6f8a482e1a3ed2e6285489d0\": rpc error: code = NotFound desc = could not find container \"59eb8924629597dfe0017423643434e1c52e925c6f8a482e1a3ed2e6285489d0\": container with ID starting with 59eb8924629597dfe0017423643434e1c52e925c6f8a482e1a3ed2e6285489d0 not found: ID does not exist"
Sep 16 19:23:58.748679 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.748657    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"]
Sep 16 19:23:58.755933 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:58.755892    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k"]
Sep 16 19:23:59.297021 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:59.296993    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" path="/var/lib/kubelet/pods/80349036-559d-40fb-8ce5-29d1d80b4d7a/volumes"
Sep 16 19:23:59.702512 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:23:59.702488    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:24:29.706785 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:29.706750    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:24:38.022653 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:38.022617    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"]
Sep 16 19:24:38.023539 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:38.023503    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln" podUID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerName="kserve-container" containerID="cri-o://88820eb44c5517c30c4f0942daa37d46c7fe457e955084411f934cb3150aa964" gracePeriod=30
Sep 16 19:24:38.023969 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:38.023897    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln" podUID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerName="kube-rbac-proxy" containerID="cri-o://8ddf5ad314026e44099f9a83fe7fb27a691b9df3c34f6461cc60d1f609ed42d6" gracePeriod=30
Sep 16 19:24:38.917098 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:38.917071    2582 generic.go:356] "Generic (PLEG): container finished" podID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerID="8ddf5ad314026e44099f9a83fe7fb27a691b9df3c34f6461cc60d1f609ed42d6" exitCode=2
Sep 16 19:24:38.917098 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:38.917095    2582 generic.go:356] "Generic (PLEG): container finished" podID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerID="88820eb44c5517c30c4f0942daa37d46c7fe457e955084411f934cb3150aa964" exitCode=0
Sep 16 19:24:38.917338 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:38.917123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln" event={"ID":"bef4c0c0-5485-4e0b-953a-763e10070e1d","Type":"ContainerDied","Data":"8ddf5ad314026e44099f9a83fe7fb27a691b9df3c34f6461cc60d1f609ed42d6"}
Sep 16 19:24:38.917338 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:38.917166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln" event={"ID":"bef4c0c0-5485-4e0b-953a-763e10070e1d","Type":"ContainerDied","Data":"88820eb44c5517c30c4f0942daa37d46c7fe457e955084411f934cb3150aa964"}
Sep 16 19:24:38.966740 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:38.966716    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:24:39.134290 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.134230    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef4c0c0-5485-4e0b-953a-763e10070e1d-cabundle-cert\") pod \"bef4c0c0-5485-4e0b-953a-763e10070e1d\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") "
Sep 16 19:24:39.134290 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.134283    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef4c0c0-5485-4e0b-953a-763e10070e1d-kube-rbac-proxy-sar-config\") pod \"bef4c0c0-5485-4e0b-953a-763e10070e1d\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") "
Sep 16 19:24:39.134692 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.134305    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xnkf\" (UniqueName: \"kubernetes.io/projected/bef4c0c0-5485-4e0b-953a-763e10070e1d-kube-api-access-7xnkf\") pod \"bef4c0c0-5485-4e0b-953a-763e10070e1d\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") "
Sep 16 19:24:39.134692 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.134334    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef4c0c0-5485-4e0b-953a-763e10070e1d-proxy-tls\") pod \"bef4c0c0-5485-4e0b-953a-763e10070e1d\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") "
Sep 16 19:24:39.134692 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.134386    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef4c0c0-5485-4e0b-953a-763e10070e1d-kserve-provision-location\") pod \"bef4c0c0-5485-4e0b-953a-763e10070e1d\" (UID: \"bef4c0c0-5485-4e0b-953a-763e10070e1d\") "
Sep 16 19:24:39.134692 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.134603    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef4c0c0-5485-4e0b-953a-763e10070e1d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bef4c0c0-5485-4e0b-953a-763e10070e1d" (UID: "bef4c0c0-5485-4e0b-953a-763e10070e1d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:39.134934 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.134693    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef4c0c0-5485-4e0b-953a-763e10070e1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bef4c0c0-5485-4e0b-953a-763e10070e1d" (UID: "bef4c0c0-5485-4e0b-953a-763e10070e1d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:39.134934 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.134825    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef4c0c0-5485-4e0b-953a-763e10070e1d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bef4c0c0-5485-4e0b-953a-763e10070e1d" (UID: "bef4c0c0-5485-4e0b-953a-763e10070e1d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:39.136612 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.136588    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef4c0c0-5485-4e0b-953a-763e10070e1d-kube-api-access-7xnkf" (OuterVolumeSpecName: "kube-api-access-7xnkf") pod "bef4c0c0-5485-4e0b-953a-763e10070e1d" (UID: "bef4c0c0-5485-4e0b-953a-763e10070e1d"). InnerVolumeSpecName "kube-api-access-7xnkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:39.136612 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.136593    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef4c0c0-5485-4e0b-953a-763e10070e1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bef4c0c0-5485-4e0b-953a-763e10070e1d" (UID: "bef4c0c0-5485-4e0b-953a-763e10070e1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:39.235504 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.235480    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef4c0c0-5485-4e0b-953a-763e10070e1d-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:24:39.235504 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.235503    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef4c0c0-5485-4e0b-953a-763e10070e1d-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:24:39.235636 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.235512    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef4c0c0-5485-4e0b-953a-763e10070e1d-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:24:39.235636 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.235521    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef4c0c0-5485-4e0b-953a-763e10070e1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:24:39.235636 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.235530    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xnkf\" (UniqueName: \"kubernetes.io/projected/bef4c0c0-5485-4e0b-953a-763e10070e1d-kube-api-access-7xnkf\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:24:39.923384 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.923358    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"
Sep 16 19:24:39.923546 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.923375    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln" event={"ID":"bef4c0c0-5485-4e0b-953a-763e10070e1d","Type":"ContainerDied","Data":"09d3e0d5302151ff2f786f0902aa9073f2638ca741c27115cbf5c0d90e0efc21"}
Sep 16 19:24:39.923546 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.923461    2582 scope.go:117] "RemoveContainer" containerID="8ddf5ad314026e44099f9a83fe7fb27a691b9df3c34f6461cc60d1f609ed42d6"
Sep 16 19:24:39.935440 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.935421    2582 scope.go:117] "RemoveContainer" containerID="88820eb44c5517c30c4f0942daa37d46c7fe457e955084411f934cb3150aa964"
Sep 16 19:24:39.943433 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.943412    2582 scope.go:117] "RemoveContainer" containerID="f4398b8de996c7174988c85ee658812e81710673eebb232d22d85c0510035eab"
Sep 16 19:24:39.957416 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.957395    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"]
Sep 16 19:24:39.964680 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:39.964660    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln"]
Sep 16 19:24:41.296769 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:24:41.296725    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bef4c0c0-5485-4e0b-953a-763e10070e1d" path="/var/lib/kubelet/pods/bef4c0c0-5485-4e0b-953a-763e10070e1d/volumes"
Sep 16 19:25:05.465777 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:05.465699    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:25:05.471515 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:05.471488    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:25:08.610227 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610193    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"]
Sep 16 19:25:08.610601 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610529    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerName="kserve-container"
Sep 16 19:25:08.610601 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610540    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerName="kserve-container"
Sep 16 19:25:08.610601 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610573    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerName="storage-initializer"
Sep 16 19:25:08.610601 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610580    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerName="storage-initializer"
Sep 16 19:25:08.610601 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610589    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerName="kserve-container"
Sep 16 19:25:08.610601 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610595    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerName="kserve-container"
Sep 16 19:25:08.610601 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610602    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerName="kube-rbac-proxy"
Sep 16 19:25:08.610830 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610610    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerName="kube-rbac-proxy"
Sep 16 19:25:08.610830 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610622    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerName="kube-rbac-proxy"
Sep 16 19:25:08.610830 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610630    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerName="kube-rbac-proxy"
Sep 16 19:25:08.610830 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610648    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerName="storage-initializer"
Sep 16 19:25:08.610830 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610653    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerName="storage-initializer"
Sep 16 19:25:08.610830 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610705    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerName="kserve-container"
Sep 16 19:25:08.610830 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610715    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerName="kube-rbac-proxy"
Sep 16 19:25:08.610830 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610722    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef4c0c0-5485-4e0b-953a-763e10070e1d" containerName="kserve-container"
Sep 16 19:25:08.610830 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.610727    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="80349036-559d-40fb-8ce5-29d1d80b4d7a" containerName="kube-rbac-proxy"
Sep 16 19:25:08.613917 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.613889    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:08.620211 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:25:08.620166    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 16 19:25:08.620807 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:25:08.620781    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 16 19:25:08.620893 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:25:08.620809    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"seaweedfs-s3-creds\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" type="*v1.Secret"
Sep 16 19:25:08.620893 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:25:08.620819    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" podUID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:08.621603 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:25:08.621576    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" type="*v1.ConfigMap"
Sep 16 19:25:08.621933 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:25:08.621879    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 19:25:08.622016 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:25:08.621931    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-mlflow-v2-runtime-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\"" type="*v1.Secret"
Sep 16 19:25:08.623251 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:25:08.623228    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-pv9ct\" is forbidden: User \"system:node:ip-10-0-129-50.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-129-50.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\"" type="*v1.Secret"
Sep 16 19:25:08.630140 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.630111    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"]
Sep 16 19:25:08.654760 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.654739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:08.654851 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.654769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:08.654851 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.654822    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:08.654851 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.654838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhj9j\" (UniqueName: \"kubernetes.io/projected/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-api-access-hhj9j\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:08.654980 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.654860    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:08.755626 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.755603    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:08.755758 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.755636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:08.755758 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.755655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:08.755758 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.755711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:08.755758 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.755732    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhj9j\" (UniqueName: \"kubernetes.io/projected/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-api-access-hhj9j\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:08.756080 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:08.756060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:09.427446 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:09.427418    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:25:09.437199 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:09.437169    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:09.571227 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:09.571201    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:25:09.571528 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:09.571508    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:25:09.576345 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:09.576316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:09.733629 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:09.733556    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 16 19:25:09.738490 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:09.738461    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:09.765894 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:25:09.765869    2582 projected.go:291] Couldn't get configMap kserve-ci-e2e-test/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 16 19:25:09.766017 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:25:09.765914    2582 projected.go:196] Error preparing data for projected volume kube-api-access-hhj9j for pod kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc: failed to sync configmap cache: timed out waiting for the condition
Sep 16 19:25:09.766017 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:25:09.765965    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-api-access-hhj9j podName:370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0 nodeName:}" failed. No retries permitted until 2026-09-16 19:25:10.265950833 +0000 UTC m=+2705.453326618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hhj9j" (UniqueName: "kubernetes.io/projected/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-api-access-hhj9j") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc" (UID: "370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 19:25:09.851367 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:09.851346    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:25:09.994729 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:09.994672    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:25:10.160465 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:10.160445    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:25:10.268046 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:10.267984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhj9j\" (UniqueName: \"kubernetes.io/projected/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-api-access-hhj9j\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:10.270652 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:10.270629    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhj9j\" (UniqueName: \"kubernetes.io/projected/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-api-access-hhj9j\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:10.424682 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:10.424656    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:10.713713 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:10.713655    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"]
Sep 16 19:25:10.716344 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:25:10.716323    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370f7e3b_ed18_4cb3_9718_2ffe2e12a5d0.slice/crio-d231e03448c936211a0f2392bea6776cdd8b7d398368133709b0f33798c67d83 WatchSource:0}: Error finding container d231e03448c936211a0f2392bea6776cdd8b7d398368133709b0f33798c67d83: Status 404 returned error can't find the container with id d231e03448c936211a0f2392bea6776cdd8b7d398368133709b0f33798c67d83
Sep 16 19:25:11.067883 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:11.067803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc" event={"ID":"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0","Type":"ContainerStarted","Data":"101d8f0fc702e690859e56f745b2b968c24399de4c18b24cc3bd9214eed9981b"}
Sep 16 19:25:11.067883 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:11.067864    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc" event={"ID":"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0","Type":"ContainerStarted","Data":"d231e03448c936211a0f2392bea6776cdd8b7d398368133709b0f33798c67d83"}
Sep 16 19:25:15.088420 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:15.088382    2582 generic.go:356] "Generic (PLEG): container finished" podID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerID="101d8f0fc702e690859e56f745b2b968c24399de4c18b24cc3bd9214eed9981b" exitCode=0
Sep 16 19:25:15.088831 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:15.088461    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc" event={"ID":"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0","Type":"ContainerDied","Data":"101d8f0fc702e690859e56f745b2b968c24399de4c18b24cc3bd9214eed9981b"}
Sep 16 19:25:16.095612 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:16.095573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc" event={"ID":"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0","Type":"ContainerStarted","Data":"b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163"}
Sep 16 19:25:16.096194 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:16.095649    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc" event={"ID":"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0","Type":"ContainerStarted","Data":"bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59"}
Sep 16 19:25:16.096194 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:16.095893    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:16.096194 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:16.095967    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:16.128711 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:16.128660    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc" podStartSLOduration=4.291750978 podStartE2EDuration="8.128641764s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.836890786s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:16.124621036 +0000 UTC m=+2711.311996823" watchObservedRunningTime="2026-09-16 19:25:16.128641764 +0000 UTC m=+2711.316017555"
Sep 16 19:25:22.105688 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:22.105657    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:52.109639 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:52.109605    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:58.627112 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:58.627071    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"]
Sep 16 19:25:58.627532 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:58.627301    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc" podUID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerName="kserve-container" containerID="cri-o://bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59" gracePeriod=30
Sep 16 19:25:58.627532 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:58.627392    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc" podUID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerName="kube-rbac-proxy" containerID="cri-o://b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163" gracePeriod=30
Sep 16 19:25:59.286003 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.285966    2582 generic.go:356] "Generic (PLEG): container finished" podID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerID="b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163" exitCode=2
Sep 16 19:25:59.286176 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.286037    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc" event={"ID":"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0","Type":"ContainerDied","Data":"b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163"}
Sep 16 19:25:59.688362 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.688339    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:25:59.851944 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.851867    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kserve-provision-location\") pod \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") "
Sep 16 19:25:59.851944 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.851922    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-rbac-proxy-sar-config\") pod \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") "
Sep 16 19:25:59.852103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.851953    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-proxy-tls\") pod \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") "
Sep 16 19:25:59.852103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.851980    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-cabundle-cert\") pod \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") "
Sep 16 19:25:59.852103 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.852006    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhj9j\" (UniqueName: \"kubernetes.io/projected/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-api-access-hhj9j\") pod \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\" (UID: \"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0\") "
Sep 16 19:25:59.852374 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.852344    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" (UID: "370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:59.852374 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.852357    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" (UID: "370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:59.852483 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.852374    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" (UID: "370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:59.854302 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.854279    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-api-access-hhj9j" (OuterVolumeSpecName: "kube-api-access-hhj9j") pod "370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" (UID: "370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0"). InnerVolumeSpecName "kube-api-access-hhj9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:59.854302 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.854283    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" (UID: "370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:59.952750 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.952724    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhj9j\" (UniqueName: \"kubernetes.io/projected/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-api-access-hhj9j\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:25:59.952750 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.952747    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:25:59.952886 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.952756    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:25:59.952886 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.952765    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:25:59.952886 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:25:59.952774    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:26:00.293177 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.293147    2582 generic.go:356] "Generic (PLEG): container finished" podID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerID="bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59" exitCode=0
Sep 16 19:26:00.293310 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.293199    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"
Sep 16 19:26:00.293310 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.293234    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc" event={"ID":"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0","Type":"ContainerDied","Data":"bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59"}
Sep 16 19:26:00.293310 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.293291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc" event={"ID":"370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0","Type":"ContainerDied","Data":"d231e03448c936211a0f2392bea6776cdd8b7d398368133709b0f33798c67d83"}
Sep 16 19:26:00.293417 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.293318    2582 scope.go:117] "RemoveContainer" containerID="b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163"
Sep 16 19:26:00.304938 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.304828    2582 scope.go:117] "RemoveContainer" containerID="bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59"
Sep 16 19:26:00.314824 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.314803    2582 scope.go:117] "RemoveContainer" containerID="101d8f0fc702e690859e56f745b2b968c24399de4c18b24cc3bd9214eed9981b"
Sep 16 19:26:00.321438 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.321410    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"]
Sep 16 19:26:00.323339 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.323324    2582 scope.go:117] "RemoveContainer" containerID="b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163"
Sep 16 19:26:00.323604 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:26:00.323587    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163\": container with ID starting with b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163 not found: ID does not exist" containerID="b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163"
Sep 16 19:26:00.323660 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.323611    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163"} err="failed to get container status \"b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163\": rpc error: code = NotFound desc = could not find container \"b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163\": container with ID starting with b0eb0595e119d7d26bc325da6ce5803a74d2b18c3b6db1a9713e452102f53163 not found: ID does not exist"
Sep 16 19:26:00.323660 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.323626    2582 scope.go:117] "RemoveContainer" containerID="bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59"
Sep 16 19:26:00.324031 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:26:00.324003    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59\": container with ID starting with bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59 not found: ID does not exist" containerID="bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59"
Sep 16 19:26:00.324188 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.324040    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59"} err="failed to get container status \"bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59\": rpc error: code = NotFound desc = could not find container \"bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59\": container with ID starting with bd976610fea69167a30aa566d184ffbcc32adde95a57fa9d7447252332cd4b59 not found: ID does not exist"
Sep 16 19:26:00.324188 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.324065    2582 scope.go:117] "RemoveContainer" containerID="101d8f0fc702e690859e56f745b2b968c24399de4c18b24cc3bd9214eed9981b"
Sep 16 19:26:00.324383 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:26:00.324367    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101d8f0fc702e690859e56f745b2b968c24399de4c18b24cc3bd9214eed9981b\": container with ID starting with 101d8f0fc702e690859e56f745b2b968c24399de4c18b24cc3bd9214eed9981b not found: ID does not exist" containerID="101d8f0fc702e690859e56f745b2b968c24399de4c18b24cc3bd9214eed9981b"
Sep 16 19:26:00.324431 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.324389    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101d8f0fc702e690859e56f745b2b968c24399de4c18b24cc3bd9214eed9981b"} err="failed to get container status \"101d8f0fc702e690859e56f745b2b968c24399de4c18b24cc3bd9214eed9981b\": rpc error: code = NotFound desc = could not find container \"101d8f0fc702e690859e56f745b2b968c24399de4c18b24cc3bd9214eed9981b\": container with ID starting with 101d8f0fc702e690859e56f745b2b968c24399de4c18b24cc3bd9214eed9981b not found: ID does not exist"
Sep 16 19:26:00.326799 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:00.326776    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc"]
Sep 16 19:26:01.296971 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:01.296941    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" path="/var/lib/kubelet/pods/370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0/volumes"
Sep 16 19:26:41.675668 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.675583    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"]
Sep 16 19:26:41.676243 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.676096    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerName="kube-rbac-proxy"
Sep 16 19:26:41.676243 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.676115    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerName="kube-rbac-proxy"
Sep 16 19:26:41.676243 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.676143    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerName="kserve-container"
Sep 16 19:26:41.676243 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.676151    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerName="kserve-container"
Sep 16 19:26:41.676243 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.676174    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerName="storage-initializer"
Sep 16 19:26:41.676243 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.676183    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerName="storage-initializer"
Sep 16 19:26:41.676573 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.676258    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerName="kserve-container"
Sep 16 19:26:41.676573 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.676273    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="370f7e3b-ed18-4cb3-9718-2ffe2e12a5d0" containerName="kube-rbac-proxy"
Sep 16 19:26:41.679375 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.679354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.682780 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.682761    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:26:41.683236 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.683218    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 16 19:26:41.683339 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.683263    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:26:41.683566 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.683550    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:26:41.683566 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.683559    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:26:41.683725 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.683549    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:26:41.683725 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.683583    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:26:41.693161 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.693142    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"]
Sep 16 19:26:41.759145 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.759123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.759243 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.759151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.759243 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.759167    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fded820b-93e9-4d28-8bf5-1c0beb2311f8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.759243 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.759230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghdfx\" (UniqueName: \"kubernetes.io/projected/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kube-api-access-ghdfx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.759348 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.759305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fded820b-93e9-4d28-8bf5-1c0beb2311f8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.860243 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.860218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghdfx\" (UniqueName: \"kubernetes.io/projected/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kube-api-access-ghdfx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.860341 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.860258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fded820b-93e9-4d28-8bf5-1c0beb2311f8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.860341 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.860301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.860341 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.860318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.860341 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.860335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fded820b-93e9-4d28-8bf5-1c0beb2311f8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.860766 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.860744    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.861028 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.861007    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fded820b-93e9-4d28-8bf5-1c0beb2311f8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.861114 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.861096    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.862950 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.862921    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fded820b-93e9-4d28-8bf5-1c0beb2311f8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.872409 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.872388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghdfx\" (UniqueName: \"kubernetes.io/projected/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kube-api-access-ghdfx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:41.990095 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:41.990042    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:42.270785 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:42.270761    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"]
Sep 16 19:26:42.272583 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:26:42.272558    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfded820b_93e9_4d28_8bf5_1c0beb2311f8.slice/crio-4d0d900955b01fc03f026134a005d45e3f2857f7c7d72b73bdedd3765b7a303f WatchSource:0}: Error finding container 4d0d900955b01fc03f026134a005d45e3f2857f7c7d72b73bdedd3765b7a303f: Status 404 returned error can't find the container with id 4d0d900955b01fc03f026134a005d45e3f2857f7c7d72b73bdedd3765b7a303f
Sep 16 19:26:42.474005 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:42.473974    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" event={"ID":"fded820b-93e9-4d28-8bf5-1c0beb2311f8","Type":"ContainerStarted","Data":"3a74850dfdd4559331fb8d153cae5a7c199e7e91f6f27a627f2391cab4b121ae"}
Sep 16 19:26:42.474137 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:42.474040    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" event={"ID":"fded820b-93e9-4d28-8bf5-1c0beb2311f8","Type":"ContainerStarted","Data":"4d0d900955b01fc03f026134a005d45e3f2857f7c7d72b73bdedd3765b7a303f"}
Sep 16 19:26:43.480540 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:43.480510    2582 generic.go:356] "Generic (PLEG): container finished" podID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerID="3a74850dfdd4559331fb8d153cae5a7c199e7e91f6f27a627f2391cab4b121ae" exitCode=0
Sep 16 19:26:43.480862 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:43.480591    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" event={"ID":"fded820b-93e9-4d28-8bf5-1c0beb2311f8","Type":"ContainerDied","Data":"3a74850dfdd4559331fb8d153cae5a7c199e7e91f6f27a627f2391cab4b121ae"}
Sep 16 19:26:44.488602 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:44.488564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" event={"ID":"fded820b-93e9-4d28-8bf5-1c0beb2311f8","Type":"ContainerStarted","Data":"bf5a5cc0a4fa9d31f417667343fe4b67dd17ed23434cc456b1fee2478b42b3b1"}
Sep 16 19:26:44.489120 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:44.488626    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" event={"ID":"fded820b-93e9-4d28-8bf5-1c0beb2311f8","Type":"ContainerStarted","Data":"7165b782dfaa9573e31a7fdf0d915db99a0ffa5745d4154d504d7b90eeb9c288"}
Sep 16 19:26:44.489120 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:44.488796    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:44.489120 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:44.488943    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:44.490470 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:44.490434    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:26:44.519189 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:44.519142    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podStartSLOduration=2.514639183 podStartE2EDuration="3.51912573s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.004486547s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:44.518470758 +0000 UTC m=+2799.705846546" watchObservedRunningTime="2026-09-16 19:26:44.51912573 +0000 UTC m=+2799.706501517"
Sep 16 19:26:45.492886 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:45.492847    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:26:50.497693 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:50.497661    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:26:50.498191 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:26:50.498165    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:27:00.498253 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:00.498218    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:27:10.498124 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:10.498082    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:27:20.498780 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:20.498745    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:27:29.104921 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.104884    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"]
Sep 16 19:27:29.108649 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.108633    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.112800 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.112779    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 16 19:27:29.112972 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.112957    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 16 19:27:29.113287 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.113271    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:27:29.126736 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.126709    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"]
Sep 16 19:27:29.212441 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.212415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a9fe20c-e791-440d-939c-d88cff10b9a5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.212542 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.212451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a9fe20c-e791-440d-939c-d88cff10b9a5-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.212542 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.212469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a9fe20c-e791-440d-939c-d88cff10b9a5-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.212542 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.212514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dwl4\" (UniqueName: \"kubernetes.io/projected/6a9fe20c-e791-440d-939c-d88cff10b9a5-kube-api-access-5dwl4\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.212649 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.212551    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a9fe20c-e791-440d-939c-d88cff10b9a5-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.313826 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.313806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dwl4\" (UniqueName: \"kubernetes.io/projected/6a9fe20c-e791-440d-939c-d88cff10b9a5-kube-api-access-5dwl4\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.313949 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.313836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a9fe20c-e791-440d-939c-d88cff10b9a5-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.314007 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:27:29.313967    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 16 19:27:29.314056 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:27:29.314011    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9fe20c-e791-440d-939c-d88cff10b9a5-proxy-tls podName:6a9fe20c-e791-440d-939c-d88cff10b9a5 nodeName:}" failed. No retries permitted until 2026-09-16 19:27:29.813996796 +0000 UTC m=+2845.001372562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a9fe20c-e791-440d-939c-d88cff10b9a5-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-4fvsm" (UID: "6a9fe20c-e791-440d-939c-d88cff10b9a5") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 16 19:27:29.314056 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.314023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a9fe20c-e791-440d-939c-d88cff10b9a5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.314056 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.314044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a9fe20c-e791-440d-939c-d88cff10b9a5-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.314187 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.314060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a9fe20c-e791-440d-939c-d88cff10b9a5-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.314501 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.314483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a9fe20c-e791-440d-939c-d88cff10b9a5-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.314630 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.314613    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a9fe20c-e791-440d-939c-d88cff10b9a5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.314700 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.314684    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a9fe20c-e791-440d-939c-d88cff10b9a5-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.326013 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.325990    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dwl4\" (UniqueName: \"kubernetes.io/projected/6a9fe20c-e791-440d-939c-d88cff10b9a5-kube-api-access-5dwl4\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.817037 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.817005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a9fe20c-e791-440d-939c-d88cff10b9a5-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:29.819751 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:29.819726    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a9fe20c-e791-440d-939c-d88cff10b9a5-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4fvsm\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:30.018595 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:30.018572    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:30.305295 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:27:30.305268    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9fe20c_e791_440d_939c_d88cff10b9a5.slice/crio-001faff1a51cd809912b6f77c31009f8d8d78038437d496b2b8a5cfd4e3a7ca7 WatchSource:0}: Error finding container 001faff1a51cd809912b6f77c31009f8d8d78038437d496b2b8a5cfd4e3a7ca7: Status 404 returned error can't find the container with id 001faff1a51cd809912b6f77c31009f8d8d78038437d496b2b8a5cfd4e3a7ca7
Sep 16 19:27:30.305638 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:30.305457    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"]
Sep 16 19:27:30.498629 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:30.498587    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:27:30.698735 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:30.698697    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" event={"ID":"6a9fe20c-e791-440d-939c-d88cff10b9a5","Type":"ContainerStarted","Data":"b96acb4912bcc9113af0b643a93bda10d19238ce8c983f63427b187249795461"}
Sep 16 19:27:30.698916 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:30.698804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" event={"ID":"6a9fe20c-e791-440d-939c-d88cff10b9a5","Type":"ContainerStarted","Data":"001faff1a51cd809912b6f77c31009f8d8d78038437d496b2b8a5cfd4e3a7ca7"}
Sep 16 19:27:35.725725 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:35.725692    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerID="b96acb4912bcc9113af0b643a93bda10d19238ce8c983f63427b187249795461" exitCode=0
Sep 16 19:27:35.726110 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:35.725764    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" event={"ID":"6a9fe20c-e791-440d-939c-d88cff10b9a5","Type":"ContainerDied","Data":"b96acb4912bcc9113af0b643a93bda10d19238ce8c983f63427b187249795461"}
Sep 16 19:27:35.726981 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:35.726964    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:27:40.498571 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:40.498528    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:27:47.795403 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:47.795365    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" event={"ID":"6a9fe20c-e791-440d-939c-d88cff10b9a5","Type":"ContainerStarted","Data":"c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51"}
Sep 16 19:27:48.802236 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:48.802202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" event={"ID":"6a9fe20c-e791-440d-939c-d88cff10b9a5","Type":"ContainerStarted","Data":"4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3"}
Sep 16 19:27:48.802661 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:48.802266    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:48.802661 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:48.802290    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:48.803552 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:48.803525    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:27:48.839990 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:48.839937    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" podStartSLOduration=3.132236482 podStartE2EDuration="19.839901014s" totalImagesPullingTime="11.98421784s" totalInitContainerRuntime="4.723446692s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:48.836589612 +0000 UTC m=+2864.023965399" watchObservedRunningTime="2026-09-16 19:27:48.839901014 +0000 UTC m=+2864.027276801"
Sep 16 19:27:49.807641 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:49.807594    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:27:50.499058 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:50.499015    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:27:54.812644 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:54.812616    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:27:54.813131 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:27:54.813105    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:28:00.292094 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:00.292019    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:28:01.615532 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:01.615498    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"]
Sep 16 19:28:01.865084 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:01.865032    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container" containerID="cri-o://7165b782dfaa9573e31a7fdf0d915db99a0ffa5745d4154d504d7b90eeb9c288" gracePeriod=30
Sep 16 19:28:01.865084 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:01.865062    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kube-rbac-proxy" containerID="cri-o://bf5a5cc0a4fa9d31f417667343fe4b67dd17ed23434cc456b1fee2478b42b3b1" gracePeriod=30
Sep 16 19:28:02.871840 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:02.871808    2582 generic.go:356] "Generic (PLEG): container finished" podID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerID="bf5a5cc0a4fa9d31f417667343fe4b67dd17ed23434cc456b1fee2478b42b3b1" exitCode=2
Sep 16 19:28:02.872232 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:02.871873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" event={"ID":"fded820b-93e9-4d28-8bf5-1c0beb2311f8","Type":"ContainerDied","Data":"bf5a5cc0a4fa9d31f417667343fe4b67dd17ed23434cc456b1fee2478b42b3b1"}
Sep 16 19:28:04.813157 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:04.813118    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:28:05.493559 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:05.493528    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Sep 16 19:28:05.887762 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:05.887730    2582 generic.go:356] "Generic (PLEG): container finished" podID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerID="7165b782dfaa9573e31a7fdf0d915db99a0ffa5745d4154d504d7b90eeb9c288" exitCode=0
Sep 16 19:28:05.888054 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:05.887794    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" event={"ID":"fded820b-93e9-4d28-8bf5-1c0beb2311f8","Type":"ContainerDied","Data":"7165b782dfaa9573e31a7fdf0d915db99a0ffa5745d4154d504d7b90eeb9c288"}
Sep 16 19:28:06.006085 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.006064    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:28:06.115866 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.115805    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghdfx\" (UniqueName: \"kubernetes.io/projected/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kube-api-access-ghdfx\") pod \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") "
Sep 16 19:28:06.115866 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.115841    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fded820b-93e9-4d28-8bf5-1c0beb2311f8-cabundle-cert\") pod \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") "
Sep 16 19:28:06.116105 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.115880    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kube-rbac-proxy-sar-config\") pod \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") "
Sep 16 19:28:06.116105 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.115978    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fded820b-93e9-4d28-8bf5-1c0beb2311f8-proxy-tls\") pod \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") "
Sep 16 19:28:06.116105 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.116008    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kserve-provision-location\") pod \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\" (UID: \"fded820b-93e9-4d28-8bf5-1c0beb2311f8\") "
Sep 16 19:28:06.116275 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.116253    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fded820b-93e9-4d28-8bf5-1c0beb2311f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fded820b-93e9-4d28-8bf5-1c0beb2311f8" (UID: "fded820b-93e9-4d28-8bf5-1c0beb2311f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:06.116341 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.116322    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fded820b-93e9-4d28-8bf5-1c0beb2311f8-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:28:06.116341 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.116320    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fded820b-93e9-4d28-8bf5-1c0beb2311f8" (UID: "fded820b-93e9-4d28-8bf5-1c0beb2311f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:06.116341 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.116328    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fded820b-93e9-4d28-8bf5-1c0beb2311f8" (UID: "fded820b-93e9-4d28-8bf5-1c0beb2311f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:06.118283 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.118262    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kube-api-access-ghdfx" (OuterVolumeSpecName: "kube-api-access-ghdfx") pod "fded820b-93e9-4d28-8bf5-1c0beb2311f8" (UID: "fded820b-93e9-4d28-8bf5-1c0beb2311f8"). InnerVolumeSpecName "kube-api-access-ghdfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:06.118364 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.118305    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fded820b-93e9-4d28-8bf5-1c0beb2311f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fded820b-93e9-4d28-8bf5-1c0beb2311f8" (UID: "fded820b-93e9-4d28-8bf5-1c0beb2311f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:06.216989 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.216965    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fded820b-93e9-4d28-8bf5-1c0beb2311f8-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:28:06.216989 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.216988    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:28:06.217110 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.216997    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghdfx\" (UniqueName: \"kubernetes.io/projected/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kube-api-access-ghdfx\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:28:06.217110 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.217006    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fded820b-93e9-4d28-8bf5-1c0beb2311f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:28:06.894842 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.894815    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"
Sep 16 19:28:06.895248 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.894812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4" event={"ID":"fded820b-93e9-4d28-8bf5-1c0beb2311f8","Type":"ContainerDied","Data":"4d0d900955b01fc03f026134a005d45e3f2857f7c7d72b73bdedd3765b7a303f"}
Sep 16 19:28:06.895248 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.894995    2582 scope.go:117] "RemoveContainer" containerID="bf5a5cc0a4fa9d31f417667343fe4b67dd17ed23434cc456b1fee2478b42b3b1"
Sep 16 19:28:06.904827 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.904810    2582 scope.go:117] "RemoveContainer" containerID="7165b782dfaa9573e31a7fdf0d915db99a0ffa5745d4154d504d7b90eeb9c288"
Sep 16 19:28:06.912263 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.912246    2582 scope.go:117] "RemoveContainer" containerID="3a74850dfdd4559331fb8d153cae5a7c199e7e91f6f27a627f2391cab4b121ae"
Sep 16 19:28:06.920422 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.920399    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"]
Sep 16 19:28:06.924567 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:06.924546    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4"]
Sep 16 19:28:07.296369 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:07.296345    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" path="/var/lib/kubelet/pods/fded820b-93e9-4d28-8bf5-1c0beb2311f8/volumes"
Sep 16 19:28:14.814072 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:14.814028    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:28:24.813084 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:24.813046    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:28:34.814220 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:34.814179    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:28:40.187557 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:40.187522    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"]
Sep 16 19:28:40.188005 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:40.187759    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kserve-container" containerID="cri-o://c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51" gracePeriod=30
Sep 16 19:28:40.188005 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:40.187804    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kube-rbac-proxy" containerID="cri-o://4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3" gracePeriod=30
Sep 16 19:28:41.046967 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:41.046933    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerID="4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3" exitCode=2
Sep 16 19:28:41.047136 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:41.046996    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" event={"ID":"6a9fe20c-e791-440d-939c-d88cff10b9a5","Type":"ContainerDied","Data":"4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3"}
Sep 16 19:28:42.525599 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.525570    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:28:42.680852 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.680825    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a9fe20c-e791-440d-939c-d88cff10b9a5-kube-rbac-proxy-sar-config\") pod \"6a9fe20c-e791-440d-939c-d88cff10b9a5\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") "
Sep 16 19:28:42.680852 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.680857    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dwl4\" (UniqueName: \"kubernetes.io/projected/6a9fe20c-e791-440d-939c-d88cff10b9a5-kube-api-access-5dwl4\") pod \"6a9fe20c-e791-440d-939c-d88cff10b9a5\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") "
Sep 16 19:28:42.681093 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.680900    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a9fe20c-e791-440d-939c-d88cff10b9a5-kserve-provision-location\") pod \"6a9fe20c-e791-440d-939c-d88cff10b9a5\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") "
Sep 16 19:28:42.681093 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.680958    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a9fe20c-e791-440d-939c-d88cff10b9a5-cabundle-cert\") pod \"6a9fe20c-e791-440d-939c-d88cff10b9a5\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") "
Sep 16 19:28:42.681213 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.681165    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a9fe20c-e791-440d-939c-d88cff10b9a5-proxy-tls\") pod \"6a9fe20c-e791-440d-939c-d88cff10b9a5\" (UID: \"6a9fe20c-e791-440d-939c-d88cff10b9a5\") "
Sep 16 19:28:42.681288 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.681260    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9fe20c-e791-440d-939c-d88cff10b9a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a9fe20c-e791-440d-939c-d88cff10b9a5" (UID: "6a9fe20c-e791-440d-939c-d88cff10b9a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:42.681367 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.681293    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9fe20c-e791-440d-939c-d88cff10b9a5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a9fe20c-e791-440d-939c-d88cff10b9a5" (UID: "6a9fe20c-e791-440d-939c-d88cff10b9a5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:42.681499 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.681480    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a9fe20c-e791-440d-939c-d88cff10b9a5-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:28:42.681499 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.681502    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a9fe20c-e791-440d-939c-d88cff10b9a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:28:42.683286 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.683257    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9fe20c-e791-440d-939c-d88cff10b9a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a9fe20c-e791-440d-939c-d88cff10b9a5" (UID: "6a9fe20c-e791-440d-939c-d88cff10b9a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:42.683470 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.683449    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9fe20c-e791-440d-939c-d88cff10b9a5-kube-api-access-5dwl4" (OuterVolumeSpecName: "kube-api-access-5dwl4") pod "6a9fe20c-e791-440d-939c-d88cff10b9a5" (UID: "6a9fe20c-e791-440d-939c-d88cff10b9a5"). InnerVolumeSpecName "kube-api-access-5dwl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:42.690645 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.690623    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9fe20c-e791-440d-939c-d88cff10b9a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a9fe20c-e791-440d-939c-d88cff10b9a5" (UID: "6a9fe20c-e791-440d-939c-d88cff10b9a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:42.782234 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.782213    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a9fe20c-e791-440d-939c-d88cff10b9a5-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:28:42.782234 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.782233    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dwl4\" (UniqueName: \"kubernetes.io/projected/6a9fe20c-e791-440d-939c-d88cff10b9a5-kube-api-access-5dwl4\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:28:42.782376 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:42.782248    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a9fe20c-e791-440d-939c-d88cff10b9a5-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:28:43.058270 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.058207    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerID="c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51" exitCode=0
Sep 16 19:28:43.058372 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.058266    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"
Sep 16 19:28:43.058372 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.058300    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" event={"ID":"6a9fe20c-e791-440d-939c-d88cff10b9a5","Type":"ContainerDied","Data":"c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51"}
Sep 16 19:28:43.058372 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.058364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm" event={"ID":"6a9fe20c-e791-440d-939c-d88cff10b9a5","Type":"ContainerDied","Data":"001faff1a51cd809912b6f77c31009f8d8d78038437d496b2b8a5cfd4e3a7ca7"}
Sep 16 19:28:43.058506 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.058395    2582 scope.go:117] "RemoveContainer" containerID="4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3"
Sep 16 19:28:43.067203 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.067186    2582 scope.go:117] "RemoveContainer" containerID="c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51"
Sep 16 19:28:43.075124 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.075106    2582 scope.go:117] "RemoveContainer" containerID="b96acb4912bcc9113af0b643a93bda10d19238ce8c983f63427b187249795461"
Sep 16 19:28:43.084417 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.084392    2582 scope.go:117] "RemoveContainer" containerID="4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3"
Sep 16 19:28:43.084799 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:28:43.084760    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3\": container with ID starting with 4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3 not found: ID does not exist" containerID="4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3"
Sep 16 19:28:43.084895 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.084808    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3"} err="failed to get container status \"4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3\": rpc error: code = NotFound desc = could not find container \"4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3\": container with ID starting with 4f21f10fe9744fac751c24df2d8eff0d674c2a9b043d9db863466c4be347b3f3 not found: ID does not exist"
Sep 16 19:28:43.084895 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.084834    2582 scope.go:117] "RemoveContainer" containerID="c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51"
Sep 16 19:28:43.085325 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:28:43.085253    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51\": container with ID starting with c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51 not found: ID does not exist" containerID="c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51"
Sep 16 19:28:43.085325 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.085288    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51"} err="failed to get container status \"c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51\": rpc error: code = NotFound desc = could not find container \"c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51\": container with ID starting with c323b43503783a0f42cf48b41b3aad62b3fb29de88fca4098c1c3576ce59aa51 not found: ID does not exist"
Sep 16 19:28:43.085537 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.085334    2582 scope.go:117] "RemoveContainer" containerID="b96acb4912bcc9113af0b643a93bda10d19238ce8c983f63427b187249795461"
Sep 16 19:28:43.085537 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.085495    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"]
Sep 16 19:28:43.089579 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:28:43.089552    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96acb4912bcc9113af0b643a93bda10d19238ce8c983f63427b187249795461\": container with ID starting with b96acb4912bcc9113af0b643a93bda10d19238ce8c983f63427b187249795461 not found: ID does not exist" containerID="b96acb4912bcc9113af0b643a93bda10d19238ce8c983f63427b187249795461"
Sep 16 19:28:43.089674 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.089582    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96acb4912bcc9113af0b643a93bda10d19238ce8c983f63427b187249795461"} err="failed to get container status \"b96acb4912bcc9113af0b643a93bda10d19238ce8c983f63427b187249795461\": rpc error: code = NotFound desc = could not find container \"b96acb4912bcc9113af0b643a93bda10d19238ce8c983f63427b187249795461\": container with ID starting with b96acb4912bcc9113af0b643a93bda10d19238ce8c983f63427b187249795461 not found: ID does not exist"
Sep 16 19:28:43.090844 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.090824    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm"]
Sep 16 19:28:43.296490 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:28:43.296460    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" path="/var/lib/kubelet/pods/6a9fe20c-e791-440d-939c-d88cff10b9a5/volumes"
Sep 16 19:29:45.098538 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.098441    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"]
Sep 16 19:29:45.099115 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.098973    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="storage-initializer"
Sep 16 19:29:45.099115 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.098991    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="storage-initializer"
Sep 16 19:29:45.099115 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099035    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kube-rbac-proxy"
Sep 16 19:29:45.099115 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099044    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kube-rbac-proxy"
Sep 16 19:29:45.099115 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099056    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container"
Sep 16 19:29:45.099115 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099065    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container"
Sep 16 19:29:45.099115 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099076    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kube-rbac-proxy"
Sep 16 19:29:45.099115 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099085    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kube-rbac-proxy"
Sep 16 19:29:45.099115 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099097    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="storage-initializer"
Sep 16 19:29:45.099115 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099106    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="storage-initializer"
Sep 16 19:29:45.099115 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099117    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kserve-container"
Sep 16 19:29:45.099711 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099125    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kserve-container"
Sep 16 19:29:45.099711 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099198    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kube-rbac-proxy"
Sep 16 19:29:45.099711 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099210    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kube-rbac-proxy"
Sep 16 19:29:45.099711 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099225    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fded820b-93e9-4d28-8bf5-1c0beb2311f8" containerName="kserve-container"
Sep 16 19:29:45.099711 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.099234    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a9fe20c-e791-440d-939c-d88cff10b9a5" containerName="kserve-container"
Sep 16 19:29:45.101622 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.101601    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.107423 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.107401    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 16 19:29:45.107522 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.107418    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:29:45.107522 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.107428    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:29:45.107646 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.107449    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:29:45.108153 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.107792    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:29:45.108396 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.108372    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:29:45.116659 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.116641    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"]
Sep 16 19:29:45.230520 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.230493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0884d63a-beee-4548-8045-e5fef8f0dc9f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.230622 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.230527    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4bk9\" (UniqueName: \"kubernetes.io/projected/0884d63a-beee-4548-8045-e5fef8f0dc9f-kube-api-access-t4bk9\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.230622 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.230552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0884d63a-beee-4548-8045-e5fef8f0dc9f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.230622 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.230616    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0884d63a-beee-4548-8045-e5fef8f0dc9f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.331308 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.331273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0884d63a-beee-4548-8045-e5fef8f0dc9f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.331438 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.331360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0884d63a-beee-4548-8045-e5fef8f0dc9f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.331438 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.331403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4bk9\" (UniqueName: \"kubernetes.io/projected/0884d63a-beee-4548-8045-e5fef8f0dc9f-kube-api-access-t4bk9\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.331554 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.331534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0884d63a-beee-4548-8045-e5fef8f0dc9f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.331680 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.331650    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0884d63a-beee-4548-8045-e5fef8f0dc9f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.331806 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:29:45.331703    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 16 19:29:45.331806 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:29:45.331764    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0884d63a-beee-4548-8045-e5fef8f0dc9f-proxy-tls podName:0884d63a-beee-4548-8045-e5fef8f0dc9f nodeName:}" failed. No retries permitted until 2026-09-16 19:29:45.831749718 +0000 UTC m=+2981.019125482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0884d63a-beee-4548-8045-e5fef8f0dc9f-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh" (UID: "0884d63a-beee-4548-8045-e5fef8f0dc9f") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 16 19:29:45.332098 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.332076    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0884d63a-beee-4548-8045-e5fef8f0dc9f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.341588 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.341559    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4bk9\" (UniqueName: \"kubernetes.io/projected/0884d63a-beee-4548-8045-e5fef8f0dc9f-kube-api-access-t4bk9\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.835153 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.835121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0884d63a-beee-4548-8045-e5fef8f0dc9f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:45.837824 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:45.837792    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0884d63a-beee-4548-8045-e5fef8f0dc9f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:46.011462 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:46.011436    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:46.295845 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:46.295820    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"]
Sep 16 19:29:46.298639 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:29:46.298609    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0884d63a_beee_4548_8045_e5fef8f0dc9f.slice/crio-45ed27ac612301a53757192abc1a9ac8a8b8057308cc60b6d460f2696010dd9b WatchSource:0}: Error finding container 45ed27ac612301a53757192abc1a9ac8a8b8057308cc60b6d460f2696010dd9b: Status 404 returned error can't find the container with id 45ed27ac612301a53757192abc1a9ac8a8b8057308cc60b6d460f2696010dd9b
Sep 16 19:29:46.337134 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:46.337107    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh" event={"ID":"0884d63a-beee-4548-8045-e5fef8f0dc9f","Type":"ContainerStarted","Data":"45ed27ac612301a53757192abc1a9ac8a8b8057308cc60b6d460f2696010dd9b"}
Sep 16 19:29:47.342625 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:47.342590    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh" event={"ID":"0884d63a-beee-4548-8045-e5fef8f0dc9f","Type":"ContainerStarted","Data":"89f820f16407f227dc621dab2d26cc6acfd15c38f2f56bd9a7f8935c507ee6f0"}
Sep 16 19:29:51.362738 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:51.362716    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh_0884d63a-beee-4548-8045-e5fef8f0dc9f/storage-initializer/0.log"
Sep 16 19:29:51.363128 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:51.362751    2582 generic.go:356] "Generic (PLEG): container finished" podID="0884d63a-beee-4548-8045-e5fef8f0dc9f" containerID="89f820f16407f227dc621dab2d26cc6acfd15c38f2f56bd9a7f8935c507ee6f0" exitCode=1
Sep 16 19:29:51.363128 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:51.362823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh" event={"ID":"0884d63a-beee-4548-8045-e5fef8f0dc9f","Type":"ContainerDied","Data":"89f820f16407f227dc621dab2d26cc6acfd15c38f2f56bd9a7f8935c507ee6f0"}
Sep 16 19:29:52.369952 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:52.369897    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh_0884d63a-beee-4548-8045-e5fef8f0dc9f/storage-initializer/0.log"
Sep 16 19:29:52.370346 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:52.370046    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh" event={"ID":"0884d63a-beee-4548-8045-e5fef8f0dc9f","Type":"ContainerStarted","Data":"8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615"}
Sep 16 19:29:55.075936 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.075891    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"]
Sep 16 19:29:55.076386 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.076108    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh" podUID="0884d63a-beee-4548-8045-e5fef8f0dc9f" containerName="storage-initializer" containerID="cri-o://8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615" gracePeriod=30
Sep 16 19:29:55.805970 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.805949    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh_0884d63a-beee-4548-8045-e5fef8f0dc9f/storage-initializer/1.log"
Sep 16 19:29:55.806384 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.806367    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh_0884d63a-beee-4548-8045-e5fef8f0dc9f/storage-initializer/0.log"
Sep 16 19:29:55.808587 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.808572    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:55.915596 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.915534    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0884d63a-beee-4548-8045-e5fef8f0dc9f-kube-rbac-proxy-sar-config\") pod \"0884d63a-beee-4548-8045-e5fef8f0dc9f\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") "
Sep 16 19:29:55.915596 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.915577    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0884d63a-beee-4548-8045-e5fef8f0dc9f-proxy-tls\") pod \"0884d63a-beee-4548-8045-e5fef8f0dc9f\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") "
Sep 16 19:29:55.915730 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.915635    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4bk9\" (UniqueName: \"kubernetes.io/projected/0884d63a-beee-4548-8045-e5fef8f0dc9f-kube-api-access-t4bk9\") pod \"0884d63a-beee-4548-8045-e5fef8f0dc9f\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") "
Sep 16 19:29:55.915730 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.915685    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0884d63a-beee-4548-8045-e5fef8f0dc9f-kserve-provision-location\") pod \"0884d63a-beee-4548-8045-e5fef8f0dc9f\" (UID: \"0884d63a-beee-4548-8045-e5fef8f0dc9f\") "
Sep 16 19:29:55.916027 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.915934    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0884d63a-beee-4548-8045-e5fef8f0dc9f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0884d63a-beee-4548-8045-e5fef8f0dc9f" (UID: "0884d63a-beee-4548-8045-e5fef8f0dc9f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:55.916027 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.916008    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0884d63a-beee-4548-8045-e5fef8f0dc9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0884d63a-beee-4548-8045-e5fef8f0dc9f" (UID: "0884d63a-beee-4548-8045-e5fef8f0dc9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:55.917770 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.917752    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0884d63a-beee-4548-8045-e5fef8f0dc9f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0884d63a-beee-4548-8045-e5fef8f0dc9f" (UID: "0884d63a-beee-4548-8045-e5fef8f0dc9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:55.917859 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:55.917843    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0884d63a-beee-4548-8045-e5fef8f0dc9f-kube-api-access-t4bk9" (OuterVolumeSpecName: "kube-api-access-t4bk9") pod "0884d63a-beee-4548-8045-e5fef8f0dc9f" (UID: "0884d63a-beee-4548-8045-e5fef8f0dc9f"). InnerVolumeSpecName "kube-api-access-t4bk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:56.016788 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.016761    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4bk9\" (UniqueName: \"kubernetes.io/projected/0884d63a-beee-4548-8045-e5fef8f0dc9f-kube-api-access-t4bk9\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:29:56.016788 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.016782    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0884d63a-beee-4548-8045-e5fef8f0dc9f-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:29:56.016979 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.016792    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0884d63a-beee-4548-8045-e5fef8f0dc9f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:29:56.016979 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.016802    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0884d63a-beee-4548-8045-e5fef8f0dc9f-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:29:56.391106 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.391077    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh_0884d63a-beee-4548-8045-e5fef8f0dc9f/storage-initializer/1.log"
Sep 16 19:29:56.391484 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.391469    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh_0884d63a-beee-4548-8045-e5fef8f0dc9f/storage-initializer/0.log"
Sep 16 19:29:56.391543 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.391510    2582 generic.go:356] "Generic (PLEG): container finished" podID="0884d63a-beee-4548-8045-e5fef8f0dc9f" containerID="8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615" exitCode=1
Sep 16 19:29:56.391597 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.391537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh" event={"ID":"0884d63a-beee-4548-8045-e5fef8f0dc9f","Type":"ContainerDied","Data":"8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615"}
Sep 16 19:29:56.391597 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.391582    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"
Sep 16 19:29:56.391597 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.391593    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh" event={"ID":"0884d63a-beee-4548-8045-e5fef8f0dc9f","Type":"ContainerDied","Data":"45ed27ac612301a53757192abc1a9ac8a8b8057308cc60b6d460f2696010dd9b"}
Sep 16 19:29:56.391760 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.391620    2582 scope.go:117] "RemoveContainer" containerID="8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615"
Sep 16 19:29:56.401186 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.401167    2582 scope.go:117] "RemoveContainer" containerID="89f820f16407f227dc621dab2d26cc6acfd15c38f2f56bd9a7f8935c507ee6f0"
Sep 16 19:29:56.408739 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.408716    2582 scope.go:117] "RemoveContainer" containerID="8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615"
Sep 16 19:29:56.409010 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:29:56.408991    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615\": container with ID starting with 8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615 not found: ID does not exist" containerID="8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615"
Sep 16 19:29:56.409081 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.409017    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615"} err="failed to get container status \"8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615\": rpc error: code = NotFound desc = could not find container \"8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615\": container with ID starting with 8bba39a7d319eb7197a814f88485739ed163a3851e19c3573645dcdf58ead615 not found: ID does not exist"
Sep 16 19:29:56.409081 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.409033    2582 scope.go:117] "RemoveContainer" containerID="89f820f16407f227dc621dab2d26cc6acfd15c38f2f56bd9a7f8935c507ee6f0"
Sep 16 19:29:56.409252 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:29:56.409239    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f820f16407f227dc621dab2d26cc6acfd15c38f2f56bd9a7f8935c507ee6f0\": container with ID starting with 89f820f16407f227dc621dab2d26cc6acfd15c38f2f56bd9a7f8935c507ee6f0 not found: ID does not exist" containerID="89f820f16407f227dc621dab2d26cc6acfd15c38f2f56bd9a7f8935c507ee6f0"
Sep 16 19:29:56.409303 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.409254    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f820f16407f227dc621dab2d26cc6acfd15c38f2f56bd9a7f8935c507ee6f0"} err="failed to get container status \"89f820f16407f227dc621dab2d26cc6acfd15c38f2f56bd9a7f8935c507ee6f0\": rpc error: code = NotFound desc = could not find container \"89f820f16407f227dc621dab2d26cc6acfd15c38f2f56bd9a7f8935c507ee6f0\": container with ID starting with 89f820f16407f227dc621dab2d26cc6acfd15c38f2f56bd9a7f8935c507ee6f0 not found: ID does not exist"
Sep 16 19:29:56.443972 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.443948    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"]
Sep 16 19:29:56.447020 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:56.447001    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh"]
Sep 16 19:29:57.296838 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:29:57.296804    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0884d63a-beee-4548-8045-e5fef8f0dc9f" path="/var/lib/kubelet/pods/0884d63a-beee-4548-8045-e5fef8f0dc9f/volumes"
Sep 16 19:30:05.499899 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:30:05.499869    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:30:05.506622 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:30:05.506600    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:34:40.030211 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.030183    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"]
Sep 16 19:34:40.031157 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.031120    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0884d63a-beee-4548-8045-e5fef8f0dc9f" containerName="storage-initializer"
Sep 16 19:34:40.031157 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.031148    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0884d63a-beee-4548-8045-e5fef8f0dc9f" containerName="storage-initializer"
Sep 16 19:34:40.031400 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.031385    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0884d63a-beee-4548-8045-e5fef8f0dc9f" containerName="storage-initializer"
Sep 16 19:34:40.031567 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.031555    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0884d63a-beee-4548-8045-e5fef8f0dc9f" containerName="storage-initializer"
Sep 16 19:34:40.031637 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.031569    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0884d63a-beee-4548-8045-e5fef8f0dc9f" containerName="storage-initializer"
Sep 16 19:34:40.031742 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.031729    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0884d63a-beee-4548-8045-e5fef8f0dc9f" containerName="storage-initializer"
Sep 16 19:34:40.036014 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.035968    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.038875 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.038856    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-63d815\""
Sep 16 19:34:40.039225 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.039203    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-63d815-kube-rbac-proxy-sar-config\""
Sep 16 19:34:40.039342 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.039293    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:34:40.039342 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.039327    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-63d815-dockercfg-4j87p\""
Sep 16 19:34:40.039792 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.039768    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-63d815-predictor-serving-cert\""
Sep 16 19:34:40.039900 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.039802    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:34:40.040990 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.040974    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:34:40.049261 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.049240    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"]
Sep 16 19:34:40.130186 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.130164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21f59e76-7469-442f-9db1-690c88bcf02d-proxy-tls\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.130318 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.130193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21f59e76-7469-442f-9db1-690c88bcf02d-cabundle-cert\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.130318 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.130237    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21f59e76-7469-442f-9db1-690c88bcf02d-kserve-provision-location\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.130448 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.130317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slx2f\" (UniqueName: \"kubernetes.io/projected/21f59e76-7469-442f-9db1-690c88bcf02d-kube-api-access-slx2f\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.130448 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.130342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21f59e76-7469-442f-9db1-690c88bcf02d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.231519 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.231493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21f59e76-7469-442f-9db1-690c88bcf02d-kserve-provision-location\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.231616 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.231536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slx2f\" (UniqueName: \"kubernetes.io/projected/21f59e76-7469-442f-9db1-690c88bcf02d-kube-api-access-slx2f\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.231616 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.231567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21f59e76-7469-442f-9db1-690c88bcf02d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.231725 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.231624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21f59e76-7469-442f-9db1-690c88bcf02d-proxy-tls\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.231725 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.231651    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21f59e76-7469-442f-9db1-690c88bcf02d-cabundle-cert\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.231827 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:34:40.231763    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-63d815-predictor-serving-cert: secret "isvc-secondary-63d815-predictor-serving-cert" not found
Sep 16 19:34:40.231880 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:34:40.231846    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f59e76-7469-442f-9db1-690c88bcf02d-proxy-tls podName:21f59e76-7469-442f-9db1-690c88bcf02d nodeName:}" failed. No retries permitted until 2026-09-16 19:34:40.731823981 +0000 UTC m=+3275.919199765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/21f59e76-7469-442f-9db1-690c88bcf02d-proxy-tls") pod "isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f" (UID: "21f59e76-7469-442f-9db1-690c88bcf02d") : secret "isvc-secondary-63d815-predictor-serving-cert" not found
Sep 16 19:34:40.232040 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.232017    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21f59e76-7469-442f-9db1-690c88bcf02d-kserve-provision-location\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.232259 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.232241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21f59e76-7469-442f-9db1-690c88bcf02d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.232316 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.232300    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21f59e76-7469-442f-9db1-690c88bcf02d-cabundle-cert\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.243059 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.243038    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slx2f\" (UniqueName: \"kubernetes.io/projected/21f59e76-7469-442f-9db1-690c88bcf02d-kube-api-access-slx2f\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.735540 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.735511    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21f59e76-7469-442f-9db1-690c88bcf02d-proxy-tls\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.738285 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.738261    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21f59e76-7469-442f-9db1-690c88bcf02d-proxy-tls\") pod \"isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") " pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:40.947064 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:40.947039    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:41.231292 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:41.231262    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"]
Sep 16 19:34:41.234651 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:34:41.234624    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f59e76_7469_442f_9db1_690c88bcf02d.slice/crio-52161dc051fdf06faf6a1f10fc9bf74554cea79c16d5049b768117139d208af9 WatchSource:0}: Error finding container 52161dc051fdf06faf6a1f10fc9bf74554cea79c16d5049b768117139d208af9: Status 404 returned error can't find the container with id 52161dc051fdf06faf6a1f10fc9bf74554cea79c16d5049b768117139d208af9
Sep 16 19:34:41.682480 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:41.682451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f" event={"ID":"21f59e76-7469-442f-9db1-690c88bcf02d","Type":"ContainerStarted","Data":"7ae3e7c7a0cab94486e4b0316751297f7290282b137e808bffefea998426c738"}
Sep 16 19:34:41.682657 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:41.682509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f" event={"ID":"21f59e76-7469-442f-9db1-690c88bcf02d","Type":"ContainerStarted","Data":"52161dc051fdf06faf6a1f10fc9bf74554cea79c16d5049b768117139d208af9"}
Sep 16 19:34:46.707285 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:46.707256    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f_21f59e76-7469-442f-9db1-690c88bcf02d/storage-initializer/0.log"
Sep 16 19:34:46.707748 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:46.707298    2582 generic.go:356] "Generic (PLEG): container finished" podID="21f59e76-7469-442f-9db1-690c88bcf02d" containerID="7ae3e7c7a0cab94486e4b0316751297f7290282b137e808bffefea998426c738" exitCode=1
Sep 16 19:34:46.707748 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:46.707358    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f" event={"ID":"21f59e76-7469-442f-9db1-690c88bcf02d","Type":"ContainerDied","Data":"7ae3e7c7a0cab94486e4b0316751297f7290282b137e808bffefea998426c738"}
Sep 16 19:34:47.713585 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:47.713557    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f_21f59e76-7469-442f-9db1-690c88bcf02d/storage-initializer/0.log"
Sep 16 19:34:47.713961 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:47.713604    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f" event={"ID":"21f59e76-7469-442f-9db1-690c88bcf02d","Type":"ContainerStarted","Data":"8ba0787ffd3b57a39101f2bb79bedb5c12fa8bfa09613435dee05df5d886c9fe"}
Sep 16 19:34:51.733037 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:51.733011    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f_21f59e76-7469-442f-9db1-690c88bcf02d/storage-initializer/1.log"
Sep 16 19:34:51.733487 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:51.733470    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f_21f59e76-7469-442f-9db1-690c88bcf02d/storage-initializer/0.log"
Sep 16 19:34:51.733537 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:51.733508    2582 generic.go:356] "Generic (PLEG): container finished" podID="21f59e76-7469-442f-9db1-690c88bcf02d" containerID="8ba0787ffd3b57a39101f2bb79bedb5c12fa8bfa09613435dee05df5d886c9fe" exitCode=1
Sep 16 19:34:51.733574 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:51.733550    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f" event={"ID":"21f59e76-7469-442f-9db1-690c88bcf02d","Type":"ContainerDied","Data":"8ba0787ffd3b57a39101f2bb79bedb5c12fa8bfa09613435dee05df5d886c9fe"}
Sep 16 19:34:51.733617 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:51.733606    2582 scope.go:117] "RemoveContainer" containerID="7ae3e7c7a0cab94486e4b0316751297f7290282b137e808bffefea998426c738"
Sep 16 19:34:51.734004 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:51.733983    2582 scope.go:117] "RemoveContainer" containerID="7ae3e7c7a0cab94486e4b0316751297f7290282b137e808bffefea998426c738"
Sep 16 19:34:51.745249 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:34:51.745221    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f_kserve-ci-e2e-test(21f59e76-7469-442f-9db1-690c88bcf02d)\"" pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f" podUID="21f59e76-7469-442f-9db1-690c88bcf02d"
Sep 16 19:34:51.745583 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:34:51.745548    2582 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f_kserve-ci-e2e-test_21f59e76-7469-442f-9db1-690c88bcf02d_0 in pod sandbox 52161dc051fdf06faf6a1f10fc9bf74554cea79c16d5049b768117139d208af9 from index: no such id: '7ae3e7c7a0cab94486e4b0316751297f7290282b137e808bffefea998426c738'" containerID="7ae3e7c7a0cab94486e4b0316751297f7290282b137e808bffefea998426c738"
Sep 16 19:34:51.745706 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:51.745587    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae3e7c7a0cab94486e4b0316751297f7290282b137e808bffefea998426c738"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f_kserve-ci-e2e-test_21f59e76-7469-442f-9db1-690c88bcf02d_0 in pod sandbox 52161dc051fdf06faf6a1f10fc9bf74554cea79c16d5049b768117139d208af9 from index: no such id: '7ae3e7c7a0cab94486e4b0316751297f7290282b137e808bffefea998426c738'"
Sep 16 19:34:52.739426 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:52.739397    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f_21f59e76-7469-442f-9db1-690c88bcf02d/storage-initializer/1.log"
Sep 16 19:34:58.205407 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.205370    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"]
Sep 16 19:34:58.340944 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.340923    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f_21f59e76-7469-442f-9db1-690c88bcf02d/storage-initializer/1.log"
Sep 16 19:34:58.344450 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.344434    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:58.476519 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.476462    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slx2f\" (UniqueName: \"kubernetes.io/projected/21f59e76-7469-442f-9db1-690c88bcf02d-kube-api-access-slx2f\") pod \"21f59e76-7469-442f-9db1-690c88bcf02d\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") "
Sep 16 19:34:58.476519 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.476491    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21f59e76-7469-442f-9db1-690c88bcf02d-cabundle-cert\") pod \"21f59e76-7469-442f-9db1-690c88bcf02d\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") "
Sep 16 19:34:58.476519 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.476511    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21f59e76-7469-442f-9db1-690c88bcf02d-proxy-tls\") pod \"21f59e76-7469-442f-9db1-690c88bcf02d\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") "
Sep 16 19:34:58.476766 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.476528    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21f59e76-7469-442f-9db1-690c88bcf02d-kube-rbac-proxy-sar-config\") pod \"21f59e76-7469-442f-9db1-690c88bcf02d\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") "
Sep 16 19:34:58.476766 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.476598    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21f59e76-7469-442f-9db1-690c88bcf02d-kserve-provision-location\") pod \"21f59e76-7469-442f-9db1-690c88bcf02d\" (UID: \"21f59e76-7469-442f-9db1-690c88bcf02d\") "
Sep 16 19:34:58.476962 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.476934    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f59e76-7469-442f-9db1-690c88bcf02d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21f59e76-7469-442f-9db1-690c88bcf02d" (UID: "21f59e76-7469-442f-9db1-690c88bcf02d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:34:58.476962 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.476950    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f59e76-7469-442f-9db1-690c88bcf02d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21f59e76-7469-442f-9db1-690c88bcf02d" (UID: "21f59e76-7469-442f-9db1-690c88bcf02d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:58.477064 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.477012    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f59e76-7469-442f-9db1-690c88bcf02d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21f59e76-7469-442f-9db1-690c88bcf02d" (UID: "21f59e76-7469-442f-9db1-690c88bcf02d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:34:58.478633 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.478613    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f59e76-7469-442f-9db1-690c88bcf02d-kube-api-access-slx2f" (OuterVolumeSpecName: "kube-api-access-slx2f") pod "21f59e76-7469-442f-9db1-690c88bcf02d" (UID: "21f59e76-7469-442f-9db1-690c88bcf02d"). InnerVolumeSpecName "kube-api-access-slx2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:58.478756 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.478736    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f59e76-7469-442f-9db1-690c88bcf02d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21f59e76-7469-442f-9db1-690c88bcf02d" (UID: "21f59e76-7469-442f-9db1-690c88bcf02d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:58.577701 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.577677    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21f59e76-7469-442f-9db1-690c88bcf02d-kserve-provision-location\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:34:58.577701 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.577697    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slx2f\" (UniqueName: \"kubernetes.io/projected/21f59e76-7469-442f-9db1-690c88bcf02d-kube-api-access-slx2f\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:34:58.577837 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.577707    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21f59e76-7469-442f-9db1-690c88bcf02d-cabundle-cert\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:34:58.577837 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.577716    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21f59e76-7469-442f-9db1-690c88bcf02d-proxy-tls\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:34:58.577837 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.577724    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21f59e76-7469-442f-9db1-690c88bcf02d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:34:58.771999 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.771937    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f_21f59e76-7469-442f-9db1-690c88bcf02d/storage-initializer/1.log"
Sep 16 19:34:58.772101 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.772036    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"
Sep 16 19:34:58.772101 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.772078    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f" event={"ID":"21f59e76-7469-442f-9db1-690c88bcf02d","Type":"ContainerDied","Data":"52161dc051fdf06faf6a1f10fc9bf74554cea79c16d5049b768117139d208af9"}
Sep 16 19:34:58.772177 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.772163    2582 scope.go:117] "RemoveContainer" containerID="8ba0787ffd3b57a39101f2bb79bedb5c12fa8bfa09613435dee05df5d886c9fe"
Sep 16 19:34:58.832116 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.832092    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"]
Sep 16 19:34:58.833646 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:58.833627    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f"]
Sep 16 19:34:59.297128 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:34:59.297098    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21f59e76-7469-442f-9db1-690c88bcf02d" path="/var/lib/kubelet/pods/21f59e76-7469-442f-9db1-690c88bcf02d/volumes"
Sep 16 19:35:04.514043 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.514007    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k8xc4/must-gather-r9dwq"]
Sep 16 19:35:04.514421 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.514355    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f59e76-7469-442f-9db1-690c88bcf02d" containerName="storage-initializer"
Sep 16 19:35:04.514421 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.514366    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f59e76-7469-442f-9db1-690c88bcf02d" containerName="storage-initializer"
Sep 16 19:35:04.514495 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.514444    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="21f59e76-7469-442f-9db1-690c88bcf02d" containerName="storage-initializer"
Sep 16 19:35:04.514530 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.514513    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f59e76-7469-442f-9db1-690c88bcf02d" containerName="storage-initializer"
Sep 16 19:35:04.514530 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.514519    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f59e76-7469-442f-9db1-690c88bcf02d" containerName="storage-initializer"
Sep 16 19:35:04.514598 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.514563    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="21f59e76-7469-442f-9db1-690c88bcf02d" containerName="storage-initializer"
Sep 16 19:35:04.519125 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.519108    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8xc4/must-gather-r9dwq"
Sep 16 19:35:04.525859 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.525620    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k8xc4\"/\"default-dockercfg-kk2qv\""
Sep 16 19:35:04.525859 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.525684    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k8xc4\"/\"openshift-service-ca.crt\""
Sep 16 19:35:04.526200 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.526174    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k8xc4\"/\"kube-root-ca.crt\""
Sep 16 19:35:04.538058 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.535264    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8xc4/must-gather-r9dwq"]
Sep 16 19:35:04.624256 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.624230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/60b39766-f103-4dd9-b9a1-55fe678929c9-must-gather-output\") pod \"must-gather-r9dwq\" (UID: \"60b39766-f103-4dd9-b9a1-55fe678929c9\") " pod="openshift-must-gather-k8xc4/must-gather-r9dwq"
Sep 16 19:35:04.624358 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.624267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lmw\" (UniqueName: \"kubernetes.io/projected/60b39766-f103-4dd9-b9a1-55fe678929c9-kube-api-access-b5lmw\") pod \"must-gather-r9dwq\" (UID: \"60b39766-f103-4dd9-b9a1-55fe678929c9\") " pod="openshift-must-gather-k8xc4/must-gather-r9dwq"
Sep 16 19:35:04.724818 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.724794    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/60b39766-f103-4dd9-b9a1-55fe678929c9-must-gather-output\") pod \"must-gather-r9dwq\" (UID: \"60b39766-f103-4dd9-b9a1-55fe678929c9\") " pod="openshift-must-gather-k8xc4/must-gather-r9dwq"
Sep 16 19:35:04.724924 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.724827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lmw\" (UniqueName: \"kubernetes.io/projected/60b39766-f103-4dd9-b9a1-55fe678929c9-kube-api-access-b5lmw\") pod \"must-gather-r9dwq\" (UID: \"60b39766-f103-4dd9-b9a1-55fe678929c9\") " pod="openshift-must-gather-k8xc4/must-gather-r9dwq"
Sep 16 19:35:04.725122 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.725098    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/60b39766-f103-4dd9-b9a1-55fe678929c9-must-gather-output\") pod \"must-gather-r9dwq\" (UID: \"60b39766-f103-4dd9-b9a1-55fe678929c9\") " pod="openshift-must-gather-k8xc4/must-gather-r9dwq"
Sep 16 19:35:04.742468 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.742448    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lmw\" (UniqueName: \"kubernetes.io/projected/60b39766-f103-4dd9-b9a1-55fe678929c9-kube-api-access-b5lmw\") pod \"must-gather-r9dwq\" (UID: \"60b39766-f103-4dd9-b9a1-55fe678929c9\") " pod="openshift-must-gather-k8xc4/must-gather-r9dwq"
Sep 16 19:35:04.845742 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:04.845683    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8xc4/must-gather-r9dwq"
Sep 16 19:35:05.124854 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:05.124792    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8xc4/must-gather-r9dwq"]
Sep 16 19:35:05.128888 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:35:05.128864    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60b39766_f103_4dd9_b9a1_55fe678929c9.slice/crio-79d8c26f34d4b2119560ca84de8a2811fe2f9625f005d1e004a8f912b17e46ad WatchSource:0}: Error finding container 79d8c26f34d4b2119560ca84de8a2811fe2f9625f005d1e004a8f912b17e46ad: Status 404 returned error can't find the container with id 79d8c26f34d4b2119560ca84de8a2811fe2f9625f005d1e004a8f912b17e46ad
Sep 16 19:35:05.131259 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:05.131239    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:35:05.533360 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:05.533337    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:35:05.540496 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:05.540477    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:35:05.809899 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:05.809818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8xc4/must-gather-r9dwq" event={"ID":"60b39766-f103-4dd9-b9a1-55fe678929c9","Type":"ContainerStarted","Data":"79d8c26f34d4b2119560ca84de8a2811fe2f9625f005d1e004a8f912b17e46ad"}
Sep 16 19:35:11.849708 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:11.849617    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8xc4/must-gather-r9dwq" event={"ID":"60b39766-f103-4dd9-b9a1-55fe678929c9","Type":"ContainerStarted","Data":"ddda3c1b9967ffbab2ca8a7a9c5a3cb8b7710fc9b8a7d2170c89fc4548a8d952"}
Sep 16 19:35:11.849708 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:11.849663    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8xc4/must-gather-r9dwq" event={"ID":"60b39766-f103-4dd9-b9a1-55fe678929c9","Type":"ContainerStarted","Data":"5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a"}
Sep 16 19:35:11.868567 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:11.868523    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k8xc4/must-gather-r9dwq" podStartSLOduration=1.8071206279999998 podStartE2EDuration="7.868507034s" totalImagesPullingTime="6.061386406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:11.867172528 +0000 UTC m=+3307.054548318" watchObservedRunningTime="2026-09-16 19:35:11.868507034 +0000 UTC m=+3307.055882821"
Sep 16 19:35:40.991285 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:40.991251    2582 generic.go:356] "Generic (PLEG): container finished" podID="60b39766-f103-4dd9-b9a1-55fe678929c9" containerID="5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a" exitCode=0
Sep 16 19:35:40.991684 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:40.991326    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8xc4/must-gather-r9dwq" event={"ID":"60b39766-f103-4dd9-b9a1-55fe678929c9","Type":"ContainerDied","Data":"5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a"}
Sep 16 19:35:40.991684 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:40.991657    2582 scope.go:117] "RemoveContainer" containerID="5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a"
Sep 16 19:35:41.354842 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:41.354779    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-k8xc4_must-gather-r9dwq_60b39766-f103-4dd9-b9a1-55fe678929c9/gather/0.log"
Sep 16 19:35:42.111355 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.111307    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kdltt/must-gather-p8wf6"]
Sep 16 19:35:42.113889 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.113868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdltt/must-gather-p8wf6"
Sep 16 19:35:42.116362 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.116339    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kdltt\"/\"default-dockercfg-tdd8t\""
Sep 16 19:35:42.116605 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.116587    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kdltt\"/\"kube-root-ca.crt\""
Sep 16 19:35:42.116667 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.116642    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kdltt\"/\"openshift-service-ca.crt\""
Sep 16 19:35:42.129171 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.129150    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdltt/must-gather-p8wf6"]
Sep 16 19:35:42.259802 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.259770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9ba658e-dcb9-45a3-a4f2-1a5531e958c0-must-gather-output\") pod \"must-gather-p8wf6\" (UID: \"e9ba658e-dcb9-45a3-a4f2-1a5531e958c0\") " pod="openshift-must-gather-kdltt/must-gather-p8wf6"
Sep 16 19:35:42.259932 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.259810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwcxm\" (UniqueName: \"kubernetes.io/projected/e9ba658e-dcb9-45a3-a4f2-1a5531e958c0-kube-api-access-pwcxm\") pod \"must-gather-p8wf6\" (UID: \"e9ba658e-dcb9-45a3-a4f2-1a5531e958c0\") " pod="openshift-must-gather-kdltt/must-gather-p8wf6"
Sep 16 19:35:42.360451 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.360425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9ba658e-dcb9-45a3-a4f2-1a5531e958c0-must-gather-output\") pod \"must-gather-p8wf6\" (UID: \"e9ba658e-dcb9-45a3-a4f2-1a5531e958c0\") " pod="openshift-must-gather-kdltt/must-gather-p8wf6"
Sep 16 19:35:42.360572 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.360461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwcxm\" (UniqueName: \"kubernetes.io/projected/e9ba658e-dcb9-45a3-a4f2-1a5531e958c0-kube-api-access-pwcxm\") pod \"must-gather-p8wf6\" (UID: \"e9ba658e-dcb9-45a3-a4f2-1a5531e958c0\") " pod="openshift-must-gather-kdltt/must-gather-p8wf6"
Sep 16 19:35:42.360766 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.360747    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9ba658e-dcb9-45a3-a4f2-1a5531e958c0-must-gather-output\") pod \"must-gather-p8wf6\" (UID: \"e9ba658e-dcb9-45a3-a4f2-1a5531e958c0\") " pod="openshift-must-gather-kdltt/must-gather-p8wf6"
Sep 16 19:35:42.371620 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.371566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwcxm\" (UniqueName: \"kubernetes.io/projected/e9ba658e-dcb9-45a3-a4f2-1a5531e958c0-kube-api-access-pwcxm\") pod \"must-gather-p8wf6\" (UID: \"e9ba658e-dcb9-45a3-a4f2-1a5531e958c0\") " pod="openshift-must-gather-kdltt/must-gather-p8wf6"
Sep 16 19:35:42.422593 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.422571    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdltt/must-gather-p8wf6"
Sep 16 19:35:42.701223 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:42.700866    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdltt/must-gather-p8wf6"]
Sep 16 19:35:42.703715 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:35:42.703689    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9ba658e_dcb9_45a3_a4f2_1a5531e958c0.slice/crio-646042195bdc059ad3a2dfdd2513eeddca041740fa801cada87eeac098fc88fa WatchSource:0}: Error finding container 646042195bdc059ad3a2dfdd2513eeddca041740fa801cada87eeac098fc88fa: Status 404 returned error can't find the container with id 646042195bdc059ad3a2dfdd2513eeddca041740fa801cada87eeac098fc88fa
Sep 16 19:35:43.002339 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:43.002264    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdltt/must-gather-p8wf6" event={"ID":"e9ba658e-dcb9-45a3-a4f2-1a5531e958c0","Type":"ContainerStarted","Data":"646042195bdc059ad3a2dfdd2513eeddca041740fa801cada87eeac098fc88fa"}
Sep 16 19:35:44.009965 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:44.009893    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdltt/must-gather-p8wf6" event={"ID":"e9ba658e-dcb9-45a3-a4f2-1a5531e958c0","Type":"ContainerStarted","Data":"bdaa0ed3e186c852a966692e0b0251e9621b117e6aee2724eb83160db797d3c4"}
Sep 16 19:35:44.010450 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:44.009982    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdltt/must-gather-p8wf6" event={"ID":"e9ba658e-dcb9-45a3-a4f2-1a5531e958c0","Type":"ContainerStarted","Data":"99b98140543e145548923c7c4e35a35c2ec2029c1eb660c5ec58517fbbe5a739"}
Sep 16 19:35:45.448518 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:45.448488    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-stn2j_935eb2ed-1f77-4417-ad43-71cb39d08c66/global-pull-secret-syncer/0.log"
Sep 16 19:35:45.608280 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:45.608249    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bjbtm_10013ea8-6367-4299-ae77-de784d9ac6cc/konnectivity-agent/0.log"
Sep 16 19:35:45.682659 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:45.682634    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-50.ec2.internal_bcc40a76f3cacad9700ee6d1963bc3a0/haproxy/0.log"
Sep 16 19:35:46.939829 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:46.939764    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kdltt/must-gather-p8wf6" podStartSLOduration=4.13608458 podStartE2EDuration="4.939727441s" totalImagesPullingTime="803.642861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:44.037151223 +0000 UTC m=+3339.224527011" watchObservedRunningTime="2026-09-16 19:35:46.939727441 +0000 UTC m=+3342.127103228"
Sep 16 19:35:46.940601 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:46.940572    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-k8xc4/must-gather-r9dwq"]
Sep 16 19:35:46.941155 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:46.941119    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-k8xc4/must-gather-r9dwq" podUID="60b39766-f103-4dd9-b9a1-55fe678929c9" containerName="copy" containerID="cri-o://ddda3c1b9967ffbab2ca8a7a9c5a3cb8b7710fc9b8a7d2170c89fc4548a8d952" gracePeriod=2
Sep 16 19:35:46.947463 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:46.947438    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-k8xc4/must-gather-r9dwq"]
Sep 16 19:35:47.315620 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:47.314037    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-k8xc4_must-gather-r9dwq_60b39766-f103-4dd9-b9a1-55fe678929c9/copy/0.log"
Sep 16 19:35:47.315936 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:47.315899    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8xc4/must-gather-r9dwq"
Sep 16 19:35:47.412181 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:47.412136    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5lmw\" (UniqueName: \"kubernetes.io/projected/60b39766-f103-4dd9-b9a1-55fe678929c9-kube-api-access-b5lmw\") pod \"60b39766-f103-4dd9-b9a1-55fe678929c9\" (UID: \"60b39766-f103-4dd9-b9a1-55fe678929c9\") "
Sep 16 19:35:47.412380 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:47.412288    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/60b39766-f103-4dd9-b9a1-55fe678929c9-must-gather-output\") pod \"60b39766-f103-4dd9-b9a1-55fe678929c9\" (UID: \"60b39766-f103-4dd9-b9a1-55fe678929c9\") "
Sep 16 19:35:47.415053 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:47.415018    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b39766-f103-4dd9-b9a1-55fe678929c9-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "60b39766-f103-4dd9-b9a1-55fe678929c9" (UID: "60b39766-f103-4dd9-b9a1-55fe678929c9"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:47.418406 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:47.418372    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b39766-f103-4dd9-b9a1-55fe678929c9-kube-api-access-b5lmw" (OuterVolumeSpecName: "kube-api-access-b5lmw") pod "60b39766-f103-4dd9-b9a1-55fe678929c9" (UID: "60b39766-f103-4dd9-b9a1-55fe678929c9"). InnerVolumeSpecName "kube-api-access-b5lmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:47.513496 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:47.513456    2582 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/60b39766-f103-4dd9-b9a1-55fe678929c9-must-gather-output\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:35:47.513496 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:47.513498    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5lmw\" (UniqueName: \"kubernetes.io/projected/60b39766-f103-4dd9-b9a1-55fe678929c9-kube-api-access-b5lmw\") on node \"ip-10-0-129-50.ec2.internal\" DevicePath \"\""
Sep 16 19:35:48.044195 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:48.044160    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-k8xc4_must-gather-r9dwq_60b39766-f103-4dd9-b9a1-55fe678929c9/copy/0.log"
Sep 16 19:35:48.044794 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:48.044597    2582 generic.go:356] "Generic (PLEG): container finished" podID="60b39766-f103-4dd9-b9a1-55fe678929c9" containerID="ddda3c1b9967ffbab2ca8a7a9c5a3cb8b7710fc9b8a7d2170c89fc4548a8d952" exitCode=143
Sep 16 19:35:48.044794 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:48.044724    2582 scope.go:117] "RemoveContainer" containerID="ddda3c1b9967ffbab2ca8a7a9c5a3cb8b7710fc9b8a7d2170c89fc4548a8d952"
Sep 16 19:35:48.044935 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:48.044892    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8xc4/must-gather-r9dwq"
Sep 16 19:35:48.070041 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:48.069706    2582 scope.go:117] "RemoveContainer" containerID="5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a"
Sep 16 19:35:48.095317 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:48.095028    2582 scope.go:117] "RemoveContainer" containerID="ddda3c1b9967ffbab2ca8a7a9c5a3cb8b7710fc9b8a7d2170c89fc4548a8d952"
Sep 16 19:35:48.095653 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:35:48.095608    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddda3c1b9967ffbab2ca8a7a9c5a3cb8b7710fc9b8a7d2170c89fc4548a8d952\": container with ID starting with ddda3c1b9967ffbab2ca8a7a9c5a3cb8b7710fc9b8a7d2170c89fc4548a8d952 not found: ID does not exist" containerID="ddda3c1b9967ffbab2ca8a7a9c5a3cb8b7710fc9b8a7d2170c89fc4548a8d952"
Sep 16 19:35:48.095768 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:48.095650    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddda3c1b9967ffbab2ca8a7a9c5a3cb8b7710fc9b8a7d2170c89fc4548a8d952"} err="failed to get container status \"ddda3c1b9967ffbab2ca8a7a9c5a3cb8b7710fc9b8a7d2170c89fc4548a8d952\": rpc error: code = NotFound desc = could not find container \"ddda3c1b9967ffbab2ca8a7a9c5a3cb8b7710fc9b8a7d2170c89fc4548a8d952\": container with ID starting with ddda3c1b9967ffbab2ca8a7a9c5a3cb8b7710fc9b8a7d2170c89fc4548a8d952 not found: ID does not exist"
Sep 16 19:35:48.095768 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:48.095676    2582 scope.go:117] "RemoveContainer" containerID="5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a"
Sep 16 19:35:48.096296 ip-10-0-129-50 kubenswrapper[2582]: E0916 19:35:48.096202    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a\": container with ID starting with 5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a not found: ID does not exist" containerID="5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a"
Sep 16 19:35:48.096296 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:48.096257    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a"} err="failed to get container status \"5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a\": rpc error: code = NotFound desc = could not find container \"5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a\": container with ID starting with 5b775ad27cf653f956b6383ef9d2329c235ab1914d3e35b38577d2ef45b1652a not found: ID does not exist"
Sep 16 19:35:49.301412 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:49.301382    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b39766-f103-4dd9-b9a1-55fe678929c9" path="/var/lib/kubelet/pods/60b39766-f103-4dd9-b9a1-55fe678929c9/volumes"
Sep 16 19:35:49.393856 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:49.393827    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd14d037-f63a-41a2-adaf-1acf0ccd1be4/alertmanager/0.log"
Sep 16 19:35:49.431754 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:49.431726    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd14d037-f63a-41a2-adaf-1acf0ccd1be4/config-reloader/0.log"
Sep 16 19:35:49.472198 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:49.472156    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd14d037-f63a-41a2-adaf-1acf0ccd1be4/kube-rbac-proxy-web/0.log"
Sep 16 19:35:49.548713 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:49.548683    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd14d037-f63a-41a2-adaf-1acf0ccd1be4/kube-rbac-proxy/0.log"
Sep 16 19:35:49.593109 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:49.593016    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd14d037-f63a-41a2-adaf-1acf0ccd1be4/kube-rbac-proxy-metric/0.log"
Sep 16 19:35:49.640439 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:49.640413    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd14d037-f63a-41a2-adaf-1acf0ccd1be4/prom-label-proxy/0.log"
Sep 16 19:35:49.681143 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:49.681114    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd14d037-f63a-41a2-adaf-1acf0ccd1be4/init-config-reloader/0.log"
Sep 16 19:35:49.772858 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:49.772825    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-qzjqh_02eb8316-86a6-4e1b-80b0-5554a341a8aa/kube-state-metrics/0.log"
Sep 16 19:35:49.826808 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:49.826780    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-qzjqh_02eb8316-86a6-4e1b-80b0-5554a341a8aa/kube-rbac-proxy-main/0.log"
Sep 16 19:35:49.889584 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:49.889503    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-qzjqh_02eb8316-86a6-4e1b-80b0-5554a341a8aa/kube-rbac-proxy-self/0.log"
Sep 16 19:35:49.960773 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:49.960741    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-644fdd7df9-k8t76_6638b647-14f2-4098-8ce9-06175618d722/metrics-server/0.log"
Sep 16 19:35:50.009940 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.009890    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-9gpnh_ff866591-f0fb-4b72-b107-92009c38c858/monitoring-plugin/0.log"
Sep 16 19:35:50.173389 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.173311    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w26ms_acd962b5-4f89-4adc-921f-807869ec5abd/node-exporter/0.log"
Sep 16 19:35:50.204828 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.204791    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w26ms_acd962b5-4f89-4adc-921f-807869ec5abd/kube-rbac-proxy/0.log"
Sep 16 19:35:50.240330 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.240281    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w26ms_acd962b5-4f89-4adc-921f-807869ec5abd/init-textfile/0.log"
Sep 16 19:35:50.538661 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.538610    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41c47de1-f9f0-4188-8685-c7956e7b7e25/prometheus/0.log"
Sep 16 19:35:50.564267 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.564241    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41c47de1-f9f0-4188-8685-c7956e7b7e25/config-reloader/0.log"
Sep 16 19:35:50.603453 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.603424    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41c47de1-f9f0-4188-8685-c7956e7b7e25/thanos-sidecar/0.log"
Sep 16 19:35:50.632025 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.631997    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41c47de1-f9f0-4188-8685-c7956e7b7e25/kube-rbac-proxy-web/0.log"
Sep 16 19:35:50.668447 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.668394    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41c47de1-f9f0-4188-8685-c7956e7b7e25/kube-rbac-proxy/0.log"
Sep 16 19:35:50.705370 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.705338    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41c47de1-f9f0-4188-8685-c7956e7b7e25/kube-rbac-proxy-thanos/0.log"
Sep 16 19:35:50.754639 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.754605    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41c47de1-f9f0-4188-8685-c7956e7b7e25/init-config-reloader/0.log"
Sep 16 19:35:50.795730 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.795651    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-bfqmj_4e448bd5-002d-4903-9c60-6f5077f88bd5/prometheus-operator/0.log"
Sep 16 19:35:50.834929 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.834877    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-bfqmj_4e448bd5-002d-4903-9c60-6f5077f88bd5/kube-rbac-proxy/0.log"
Sep 16 19:35:50.883451 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.883424    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-jlhdj_725304b0-f416-4a92-90ff-d1a3c9be7845/prometheus-operator-admission-webhook/0.log"
Sep 16 19:35:50.944538 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.944493    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c794674b7-jnbrv_0ba91952-12c7-4a3c-81a3-00d57a122618/telemeter-client/0.log"
Sep 16 19:35:50.990759 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:50.990733    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c794674b7-jnbrv_0ba91952-12c7-4a3c-81a3-00d57a122618/reload/0.log"
Sep 16 19:35:51.029587 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:51.029560    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c794674b7-jnbrv_0ba91952-12c7-4a3c-81a3-00d57a122618/kube-rbac-proxy/0.log"
Sep 16 19:35:51.077958 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:51.077863    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c5f686858-5wp7s_65f47594-f4f8-4a74-908b-1f7810167c04/thanos-query/0.log"
Sep 16 19:35:51.110080 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:51.110049    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c5f686858-5wp7s_65f47594-f4f8-4a74-908b-1f7810167c04/kube-rbac-proxy-web/0.log"
Sep 16 19:35:51.158343 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:51.158311    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c5f686858-5wp7s_65f47594-f4f8-4a74-908b-1f7810167c04/kube-rbac-proxy/0.log"
Sep 16 19:35:51.196071 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:51.196045    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c5f686858-5wp7s_65f47594-f4f8-4a74-908b-1f7810167c04/prom-label-proxy/0.log"
Sep 16 19:35:51.235475 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:51.235438    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c5f686858-5wp7s_65f47594-f4f8-4a74-908b-1f7810167c04/kube-rbac-proxy-rules/0.log"
Sep 16 19:35:51.272352 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:51.272320    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c5f686858-5wp7s_65f47594-f4f8-4a74-908b-1f7810167c04/kube-rbac-proxy-metrics/0.log"
Sep 16 19:35:52.500300 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:52.500269    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-xsxww_4666cfdb-3186-49c5-b678-a49c4c211bcf/networking-console-plugin/0.log"
Sep 16 19:35:53.095261 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.095228    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/1.log"
Sep 16 19:35:53.101313 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.101283    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-sjlt4_3aca151a-3031-429f-aa10-fb8d21ba5a27/console-operator/2.log"
Sep 16 19:35:53.228778 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.228743    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"]
Sep 16 19:35:53.229190 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.229172    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b39766-f103-4dd9-b9a1-55fe678929c9" containerName="copy"
Sep 16 19:35:53.229285 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.229192    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b39766-f103-4dd9-b9a1-55fe678929c9" containerName="copy"
Sep 16 19:35:53.229285 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.229237    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b39766-f103-4dd9-b9a1-55fe678929c9" containerName="gather"
Sep 16 19:35:53.229285 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.229248    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b39766-f103-4dd9-b9a1-55fe678929c9" containerName="gather"
Sep 16 19:35:53.229446 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.229327    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b39766-f103-4dd9-b9a1-55fe678929c9" containerName="copy"
Sep 16 19:35:53.229446 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.229345    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b39766-f103-4dd9-b9a1-55fe678929c9" containerName="gather"
Sep 16 19:35:53.234469 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.234445    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.249142 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.249117    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"]
Sep 16 19:35:53.371786 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.371703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/550223b4-0a93-49c9-9b0d-9867b9992c87-sys\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.371786 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.371759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/550223b4-0a93-49c9-9b0d-9867b9992c87-podres\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.372056 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.371806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m6bh\" (UniqueName: \"kubernetes.io/projected/550223b4-0a93-49c9-9b0d-9867b9992c87-kube-api-access-9m6bh\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.372056 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.371835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/550223b4-0a93-49c9-9b0d-9867b9992c87-proc\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.372056 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.371979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/550223b4-0a93-49c9-9b0d-9867b9992c87-lib-modules\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.473156 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.473107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/550223b4-0a93-49c9-9b0d-9867b9992c87-sys\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.473339 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.473162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/550223b4-0a93-49c9-9b0d-9867b9992c87-podres\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.473339 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.473214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m6bh\" (UniqueName: \"kubernetes.io/projected/550223b4-0a93-49c9-9b0d-9867b9992c87-kube-api-access-9m6bh\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.473339 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.473248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/550223b4-0a93-49c9-9b0d-9867b9992c87-proc\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.473526 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.473385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/550223b4-0a93-49c9-9b0d-9867b9992c87-lib-modules\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.473589 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.473546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/550223b4-0a93-49c9-9b0d-9867b9992c87-lib-modules\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.473649 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.473605    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/550223b4-0a93-49c9-9b0d-9867b9992c87-sys\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.473707 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.473686    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/550223b4-0a93-49c9-9b0d-9867b9992c87-podres\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.474058 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.474033    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/550223b4-0a93-49c9-9b0d-9867b9992c87-proc\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.483830 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.483808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m6bh\" (UniqueName: \"kubernetes.io/projected/550223b4-0a93-49c9-9b0d-9867b9992c87-kube-api-access-9m6bh\") pod \"perf-node-gather-daemonset-xhswc\" (UID: \"550223b4-0a93-49c9-9b0d-9867b9992c87\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.546359 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.546314    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:53.838613 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:53.838580    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"]
Sep 16 19:35:53.841701 ip-10-0-129-50 kubenswrapper[2582]: W0916 19:35:53.841667    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod550223b4_0a93_49c9_9b0d_9867b9992c87.slice/crio-18c0a25b5d95de6a890b94425eb191f27e1af370ad247bc4bc9b69e1d7dcee21 WatchSource:0}: Error finding container 18c0a25b5d95de6a890b94425eb191f27e1af370ad247bc4bc9b69e1d7dcee21: Status 404 returned error can't find the container with id 18c0a25b5d95de6a890b94425eb191f27e1af370ad247bc4bc9b69e1d7dcee21
Sep 16 19:35:54.084875 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:54.084835    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc" event={"ID":"550223b4-0a93-49c9-9b0d-9867b9992c87","Type":"ContainerStarted","Data":"03039b013debc61ac653c3217f36816fc5c6e691069204a17bae8c1ea5c4faac"}
Sep 16 19:35:54.085093 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:54.084973    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:35:54.085093 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:54.085031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc" event={"ID":"550223b4-0a93-49c9-9b0d-9867b9992c87","Type":"ContainerStarted","Data":"18c0a25b5d95de6a890b94425eb191f27e1af370ad247bc4bc9b69e1d7dcee21"}
Sep 16 19:35:54.108181 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:54.108081    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc" podStartSLOduration=1.108062121 podStartE2EDuration="1.108062121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:54.105324015 +0000 UTC m=+3349.292699803" watchObservedRunningTime="2026-09-16 19:35:54.108062121 +0000 UTC m=+3349.295437920"
Sep 16 19:35:54.180277 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:54.180250    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-z6q5q_049b76d8-852b-42f0-9720-d8bb5e59ff5f/volume-data-source-validator/0.log"
Sep 16 19:35:55.286060 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:55.286033    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hl5jx_a673e9f7-670e-4604-95f2-7aab6f511727/dns/0.log"
Sep 16 19:35:55.314030 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:55.313991    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hl5jx_a673e9f7-670e-4604-95f2-7aab6f511727/kube-rbac-proxy/0.log"
Sep 16 19:35:55.403217 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:55.403196    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z5pl8_f8eeb34d-7a85-40ac-8755-be657c455ec6/dns-node-resolver/0.log"
Sep 16 19:35:56.071670 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:56.071642    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-shplt_d37b1617-d4af-42a9-8795-2074cbf8be05/node-ca/0.log"
Sep 16 19:35:57.375960 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:57.375930    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xlrfw_a00e8866-0ae0-4245-b716-faac4ee1cd3f/serve-healthcheck-canary/0.log"
Sep 16 19:35:58.029394 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:58.029368    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2kqq_5bb3ad5a-942a-4f71-a5b4-2c387ca3d376/kube-rbac-proxy/0.log"
Sep 16 19:35:58.057286 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:58.057265    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2kqq_5bb3ad5a-942a-4f71-a5b4-2c387ca3d376/exporter/0.log"
Sep 16 19:35:58.086660 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:35:58.086640    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2kqq_5bb3ad5a-942a-4f71-a5b4-2c387ca3d376/extractor/0.log"
Sep 16 19:36:00.101057 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:00.101015    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-xhswc"
Sep 16 19:36:01.035514 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:01.035473    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-zlp5j_989138ce-8ff9-4abb-ad44-b07bfb5dcf59/s3-init/0.log"
Sep 16 19:36:01.065987 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:01.065955    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-zlp5j_989138ce-8ff9-4abb-ad44-b07bfb5dcf59/download-hf-model/0.log"
Sep 16 19:36:01.103703 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:01.103676    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-58mq6_65cc7e64-0c8d-474a-88e4-f6ca805cd69c/s3-tls-init-custom/0.log"
Sep 16 19:36:01.135272 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:01.135252    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-58mq6_65cc7e64-0c8d-474a-88e4-f6ca805cd69c/download-hf-model/0.log"
Sep 16 19:36:01.182893 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:01.182866    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-wnh9x_667cbff2-42d8-434d-8890-5ed1ae1e3d59/s3-tls-init-serving/0.log"
Sep 16 19:36:01.212220 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:01.212195    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-wnh9x_667cbff2-42d8-434d-8890-5ed1ae1e3d59/download-hf-model/0.log"
Sep 16 19:36:06.418970 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:06.418927    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-cs4bd_0114f015-bf5a-4d44-b6fd-3bb49cc8e372/migrator/0.log"
Sep 16 19:36:06.463731 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:06.463693    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-cs4bd_0114f015-bf5a-4d44-b6fd-3bb49cc8e372/graceful-termination/0.log"
Sep 16 19:36:08.192305 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:08.192280    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnnjq_78095364-693e-4375-ae1e-cd23e8a6001d/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:36:08.225062 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:08.225036    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnnjq_78095364-693e-4375-ae1e-cd23e8a6001d/egress-router-binary-copy/0.log"
Sep 16 19:36:08.253537 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:08.253513    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnnjq_78095364-693e-4375-ae1e-cd23e8a6001d/cni-plugins/0.log"
Sep 16 19:36:08.283977 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:08.283961    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnnjq_78095364-693e-4375-ae1e-cd23e8a6001d/bond-cni-plugin/0.log"
Sep 16 19:36:08.315024 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:08.314991    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnnjq_78095364-693e-4375-ae1e-cd23e8a6001d/routeoverride-cni/0.log"
Sep 16 19:36:08.343522 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:08.343502    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnnjq_78095364-693e-4375-ae1e-cd23e8a6001d/whereabouts-cni-bincopy/0.log"
Sep 16 19:36:08.370977 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:08.370958    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnnjq_78095364-693e-4375-ae1e-cd23e8a6001d/whereabouts-cni/0.log"
Sep 16 19:36:08.682817 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:08.682796    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-x4jf2_dd12d598-68f2-4545-a589-be29dda56485/kube-multus/0.log"
Sep 16 19:36:08.708710 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:08.708695    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2bzzg_47c4bc73-2eff-41ea-91fc-5acce1bcd526/network-metrics-daemon/0.log"
Sep 16 19:36:08.740734 ip-10-0-129-50 kubenswrapper[2582]: I0916 19:36:08.740706    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2bzzg_47c4bc73-2eff-41ea-91fc-5acce1bcd526/kube-rbac-proxy/0.log"