Sep 18 20:18:28.146452 ip-10-0-130-79 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 20:18:28.146464 ip-10-0-130-79 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 20:18:28.146471 ip-10-0-130-79 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 20:18:28.146688 ip-10-0-130-79 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 20:18:38.242587 ip-10-0-130-79 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 20:18:38.242601 ip-10-0-130-79 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot dbe3612dc500420f89ed6936a2e2b3c3 --
Sep 18 20:20:47.629401 ip-10-0-130-79 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 20:20:51.082012 ip-10-0-130-79 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 20:20:51.082012 ip-10-0-130-79 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 20:20:51.082012 ip-10-0-130-79 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 20:20:51.082012 ip-10-0-130-79 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 20:20:51.082012 ip-10-0-130-79 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 20:20:51.082685 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.082089    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 20:20:51.085718 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085702    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:51.085718 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085718    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085722    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085726    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085729    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085731    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085734    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085737    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085740    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085743    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085746    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085749    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085751    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085754    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085757    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085759    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085762    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085765    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085768    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085770    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085773    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:51.085778 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085775    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085778    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085780    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085783    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085787    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085789    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085792    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085795    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085797    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085800    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085802    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085805    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085808    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085810    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085813    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085817    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085819    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085822    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085824    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085826    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:51.086272 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085829    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085831    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085834    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085836    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085839    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085841    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085843    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085846    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085849    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085853    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085856    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085861    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085864    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085867    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085870    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085873    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085876    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085879    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085882    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:51.086763 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085885    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085888    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085890    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085893    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085895    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085898    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085900    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085903    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085905    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085908    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085910    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085915    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085918    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085921    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085924    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085927    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085929    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085932    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085934    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:51.087232 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085937    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085939    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085942    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085945    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085947    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085950    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085952    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085955    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085959    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085962    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085965    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085967    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085970    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085972    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085975    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085978    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085981    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085984    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085986    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:51.087690 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.085989    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086721    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086729    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086732    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086735    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086738    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086741    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086744    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086747    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086750    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086753    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086756    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086758    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086761    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086764    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086766    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086770    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086772    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086775    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086777    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:51.088152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086781    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086784    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086787    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086790    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086792    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086794    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086797    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086799    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086802    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086804    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086807    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086810    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086813    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086817    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086820    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086823    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086825    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086828    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086831    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:51.088640 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086834    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086837    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086840    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086842    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086846    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086848    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086851    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086853    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086856    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086858    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086861    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086864    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086866    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086869    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086872    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086875    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086877    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086881    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086884    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:51.089131 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086887    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086889    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086892    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086894    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086897    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086899    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086902    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086904    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086906    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086909    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086911    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086914    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086916    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086919    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086922    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086925    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086927    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086930    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086933    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086935    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:51.089601 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086938    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086940    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086942    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086945    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086947    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086950    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086952    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086955    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086957    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086960    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086962    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086964    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086967    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086969    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086973    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086977    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086979    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086982    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086985    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:51.090108 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086987    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086990    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.086992    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087360    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087365    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087368    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087370    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087373    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087376    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087379    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087381    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087385    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087388    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087390    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087393    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087395    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087398    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087400    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087402    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:51.090584 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087405    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087408    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087411    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087413    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087416    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087418    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087421    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087424    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087428    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087431    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087435    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087439    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087443    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087447    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087450    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087453    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087456    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087458    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087461    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:51.091043 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087464    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087466    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087470    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087473    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087476    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087478    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087481    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087483    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087486    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087488    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087491    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087493    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087496    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087498    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087501    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087504    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087506    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087509    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087511    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087513    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:51.091544 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087516    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087518    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087521    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087523    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087526    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087528    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087531    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087534    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087536    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087539    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087541    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087544    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087546    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087551    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087553    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087556    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087558    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087560    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087563    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087566    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:51.092019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087569    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087571    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087573    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087576    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087578    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087580    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087583    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087586    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087588    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087591    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087593    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087596    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087598    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087600    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087603    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087605    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087608    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087610    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087612    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087615    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:51.092543 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087617    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087620    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087622    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.087625    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087700    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087735    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087741    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087746    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087754    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087757    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087761    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087765    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087769    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087772    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087776    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087779    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087782    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087784    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087787    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087790    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087793    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087796    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087800    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087803    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 20:20:51.093034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087806    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087809    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087813    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087816    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087819    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087822    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087825    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087829    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087831    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087834    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087837    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087840    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087844    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087849    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087852    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087854    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087857    2583 flags.go:64] FLAG: --enable-server="true"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087860    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087864    2583 flags.go:64] FLAG: --event-burst="100"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087867    2583 flags.go:64] FLAG: --event-qps="50"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087870    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087873    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087876    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087880    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 20:20:51.093626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087883    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087886    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087889    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087892    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087895    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087898    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087901    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087904    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087906    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087909    2583 flags.go:64] FLAG: --feature-gates=""
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087913    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087916    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087918    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087921    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087924    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087927    2583 flags.go:64] FLAG: --help="false"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087930    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087933    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087935    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087938    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087942    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087946    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087949    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 20:20:51.094204 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087951    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087954    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087957    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087960    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087963    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087966    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087969    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087972    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087975    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087978    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087980    2583 flags.go:64] FLAG: --lock-file=""
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087985    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087988    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087991    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087996    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.087998    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088001    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088004    2583 flags.go:64] FLAG: --logging-format="text"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088006    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088010    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088013    2583 flags.go:64] FLAG: --manifest-url=""
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088016    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088020    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088023    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088027    2583 flags.go:64] FLAG: --max-pods="110"
Sep 18 20:20:51.095076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088030    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088032    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088035    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088038    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088041    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088045    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088048    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088055    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088058    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088061    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088064    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088067    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088072    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088075    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088078    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088081    2583 flags.go:64] FLAG: --port="10250"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088084    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088087    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a685b811d7df3e24"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088091    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088094    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088097    2583 flags.go:64] FLAG: --register-node="true"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088100    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088104    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088107    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 18 20:20:51.096183 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088109    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088112    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088115    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088134    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088137    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088140    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088143    2583 flags.go:64] FLAG: --runonce="false"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088145    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088149    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088151    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088154    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088157    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088161    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088164    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088168    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088171    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088174    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088176    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088179    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088182    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088185    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088188    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088193    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088196    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088199    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 20:20:51.097224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088204    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088208    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088211    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088214    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088217    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088220    2583 flags.go:64] FLAG: --v="2"
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088224    2583 flags.go:64] FLAG: --version="false"
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088227    2583 flags.go:64] FLAG: --vmodule=""
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088231    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.088234    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088359    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088363    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088366    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088369    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088372    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088374    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088377    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088380    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088382    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088385    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088387    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088391    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:51.098290 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088394    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088397    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088399    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088402    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088404    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088407    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088410    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088412    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088415    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088418    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088421    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088429    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088432    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088435    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088439    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088441    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088444    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088447    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088450    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088453    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:51.098876 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088455    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088458    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088460    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088463    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088466    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088469    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088473    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088477    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088481    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088484    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088487    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088491    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088493    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088496    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088499    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088502    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088505    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088507    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088510    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088512    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:51.099721 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088515    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088517    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088520    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088525    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088527    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088530    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088532    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088535    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088538    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088543    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088546    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088549    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088551    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088554    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088557    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088559    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088562    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088564    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088567    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:51.100518 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088569    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088572    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088575    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088577    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088582    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088584    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088587    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088591    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088594    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088597    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088600    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088602    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088605    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088608    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088611    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088613    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088617    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088620    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088623    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:51.101142 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088625    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088628    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088630    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088633    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088635    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088638    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088641    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088643    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.088646    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.089137    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.096712    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.096731    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.096759    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.096768    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096851    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:51.101619 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096859    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096864    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096868    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096872    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096877    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096884    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096890    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096895    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096899    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096904    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096908    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096913    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096917    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096922    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096926    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096930    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096935    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096941    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:51.102019 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096947    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096951    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096955    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096959    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096964    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096968    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096972    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096977    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096981    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096985    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096991    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.096995    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097000    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097004    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097008    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097012    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097016    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097020    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097024    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097029    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:51.102538 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097033    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097037    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097041    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097045    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097049    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097053    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097057    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097061    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097065    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097069    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097073    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097077    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097081    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097085    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097089    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097093    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097097    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097101    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097106    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:51.103074 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097110    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097114    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097138    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097143    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097149    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097153    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097157    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097161    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097165    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097169    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097173    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097177    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097181    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097184    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097189    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097193    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097200    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097205    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097209    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097214    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:51.103607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097218    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097223    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097227    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097231    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097234    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097238    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097242    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097246    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097249    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097253    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097258    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097262    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097266    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097270    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097275    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097279    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097283    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097289    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097293    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097297    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:51.104081 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097301    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.097308    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097488    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097496    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097500    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097505    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097509    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097513    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097517    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097521    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097524    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097528    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097533    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097537    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097541    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:51.105036 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097545    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097549    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097553    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097557    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097561    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097565    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097569    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097573    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097578    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097583    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097587    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097591    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097595    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097599    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097604    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097608    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097613    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097616    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097621    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097625    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:51.105704 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097629    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097632    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097636    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097640    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097644    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097648    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097654    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097661    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097665    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097669    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097673    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097676    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097681    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097685    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097689    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097693    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097697    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097701    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097706    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097710    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:51.106286 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097714    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097718    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097722    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097726    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097730    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097734    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097738    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097742    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097747    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097751    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097755    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097759    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097763    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097767    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097773    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097779    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097784    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097788    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097792    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:51.106917 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097796    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097801    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097805    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097809    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097813    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097818    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097822    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097826    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097832    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097838    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097842    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097847    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097852    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097856    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097860    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097865    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097869    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097873    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097877    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:51.107612 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097881    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097885    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097889    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097893    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097897    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097901    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097905    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:51.097909    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.097917    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.098064    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.100930    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.102158    2583 server.go:1019] "Starting client certificate rotation"
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.102263    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.102303    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 20:20:51.108144 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.104766    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 20:20:51.108847 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.108827    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 20:20:51.113426 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.113409    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 18 20:20:51.114828 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.114813    2583 log.go:25] "Validated CRI v1 image API"
Sep 18 20:20:51.116337 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.116316    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 20:20:51.119412 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.119395    2583 fs.go:135] Filesystem UUIDs: map[067f9bbe-dd79-405e-8d4a-c1c3c0a88a5f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9ad3cd7c-2279-4fac-874b-9f4c14edad7e:/dev/nvme0n1p3]
Sep 18 20:20:51.119462 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.119412    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 20:20:51.124434 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.124327    2583 manager.go:216] Machine: {Timestamp:2026-09-18 20:20:51.123598723 +0000 UTC m=+2.031501003 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100882 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec284fb002273bad0f253535ac70dbcb SystemUUID:ec284fb0-0227-3bad-0f25-3535ac70dbcb BootID:dbe3612d-c500-420f-89ed-6936a2e2b3c3 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c9:b0:f7:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c9:b0:f7:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:2c:05:9f:57:3a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 20:20:51.124434 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.124430    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 20:20:51.124544 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.124499    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 20:20:51.126090 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.126067    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 20:20:51.126242 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.126093    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-79.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 20:20:51.127222 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.127202    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 20:20:51.127222 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.127225    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 20:20:51.127285 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.127232    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 20:20:51.127285 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.127247    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 20:20:51.127341 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.127316    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 20:20:51.130216 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.130205    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 20:20:51.130327 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.130317    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 20:20:51.130362 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.130353    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 20:20:51.130390 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.130367    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 20:20:51.130390 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.130386    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 20:20:51.130458 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.130449    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 18 20:20:51.130502 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.130465    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 20:20:51.130968 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.130953    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 20:20:51.131019 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.131009    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 20:20:51.133533 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.133517    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 20:20:51.135591 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.135577    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 20:20:51.135635 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.135608    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 20:20:51.136634 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136619    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 20:20:51.136705 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136641    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 20:20:51.136705 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136650    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 20:20:51.136705 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136673    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 20:20:51.136800 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136709    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 20:20:51.136800 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136722    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 20:20:51.136800 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136740    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 20:20:51.136800 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136749    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 20:20:51.136800 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136772    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 20:20:51.136800 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136779    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 20:20:51.136800 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136788    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 20:20:51.136800 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136793    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 20:20:51.136991 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136814    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 20:20:51.136991 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.136820    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 20:20:51.138509 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.138482    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 20:20:51.140997 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.140979    2583 server.go:1268] "Started kubelet"
Sep 18 20:20:51.141522 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.141489    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 20:20:51.141882 ip-10-0-130-79 systemd[1]: Started Kubernetes Kubelet.
Sep 18 20:20:51.142838 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.141967    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 20:20:51.142838 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.142040    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 20:20:51.143290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.143159    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 20:20:51.143745 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.143498    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-79.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 20:20:51.143745 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.143588    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 20:20:51.143745 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.143680    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 20:20:51.143921 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.143839    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 20:20:51.178621 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.178590    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9qk7b"
Sep 18 20:20:51.178621 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.178617    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9qk7b"
Sep 18 20:20:51.179304 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.179275    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 20:20:51.180580 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.180566    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 20:20:51.180580 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.180576    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 20:20:51.180720 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.180697    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 20:20:51.181271 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.181253    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 20:20:51.181368 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.180376    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-79.ec2.internal.18d683d49e72550a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-79.ec2.internal,UID:ip-10-0-130-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-79.ec2.internal,},FirstTimestamp:2026-09-18 20:20:51.140949258 +0000 UTC m=+2.048851528,LastTimestamp:2026-09-18 20:20:51.140949258 +0000 UTC m=+2.048851528,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-79.ec2.internal,}"
Sep 18 20:20:51.181459 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.181369    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 20:20:51.181459 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.181386    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 20:20:51.181617 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.181602    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 20:20:51.181732 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.181716    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 20:20:51.181839 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.181825    2583 factory.go:55] Registering systemd factory
Sep 18 20:20:51.181915 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.181844    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 18 20:20:51.182089 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.182075    2583 factory.go:153] Registering CRI-O factory
Sep 18 20:20:51.182089 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.182090    2583 factory.go:223] Registration of the crio container factory successfully
Sep 18 20:20:51.182202 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.182164    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 20:20:51.182202 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.182184    2583 factory.go:103] Registering Raw factory
Sep 18 20:20:51.182202 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.182201    2583 manager.go:1196] Started watching for new ooms in manager
Sep 18 20:20:51.182812 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.182792    2583 manager.go:349] Starting recovery of all containers
Sep 18 20:20:51.193073 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.192881    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:51.193852 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.193838    2583 manager.go:354] Recovery completed
Sep 18 20:20:51.195938 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.195922    2583 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-79.ec2.internal\" not found" node="ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.198606 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.198594    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:51.200471 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.200457    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:51.200528 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.200485    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:51.200528 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.200494    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:51.201072 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.201054    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 20:20:51.201072 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.201070    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 20:20:51.201198 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.201080    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:51.201198 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.201090    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 20:20:51.201743 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.201730    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:51.201797 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.201774    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:51.201797 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.201785    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:51.202309 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.202298    2583 policy_none.go:49] "None policy: Start"
Sep 18 20:20:51.202351 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.202314    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 20:20:51.202351 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.202323    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 20:20:51.203374 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.203360    2583 policy_none.go:47] "Start"
Sep 18 20:20:51.239721 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.239704    2583 manager.go:337] "Starting Device Plugin manager"
Sep 18 20:20:51.247423 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.239751    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 20:20:51.247423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.239764    2583 server.go:87] "Starting device plugin registration server"
Sep 18 20:20:51.247423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.240039    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 20:20:51.247423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.240062    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 20:20:51.247423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.240238    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 20:20:51.247423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.240348    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 20:20:51.247423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.240379    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 20:20:51.247423 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.240890    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 20:20:51.247423 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.240945    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 20:20:51.308185 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.308151    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 20:20:51.309473 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.309448    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 20:20:51.309595 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.309506    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 20:20:51.309595 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.309537    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 20:20:51.309595 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.309582    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 20:20:51.311903 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.311878    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:51.341042 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.340972    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:51.341955 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.341939    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:51.342048 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.341974    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:51.342048 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.341989    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:51.342048 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.342019    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.350556 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.350540    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.350634 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.350563    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-79.ec2.internal\": node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 20:20:51.365789 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.365768    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 20:20:51.410601 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.410571    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"]
Sep 18 20:20:51.410701 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.410677    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:51.412158 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.412144    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:51.412231 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.412177    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:51.412231 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.412193    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:51.413850 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.413834    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:51.413967 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.413952    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.414016 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.413981    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:51.414575 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.414558    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:51.414575 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.414568    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:51.414683 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.414588    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:51.414683 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.414593    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:51.414683 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.414598    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:51.414683 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.414606    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:51.415960 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.415943    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.416045 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.415966    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:51.416564 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.416548    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:51.416627 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.416578    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:51.416627 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.416592    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:51.439754 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.439731    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-79.ec2.internal\" not found" node="ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.444232 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.444216    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-79.ec2.internal\" not found" node="ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.466614 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.466596    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 20:20:51.483682 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.483663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f016832ad3a15792c6b1a23ee96cdaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"2f016832ad3a15792c6b1a23ee96cdaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.483731 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.483687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc3d8b9722788fbfdcb8bc7f7b68e6ec-config\") pod \"kube-apiserver-proxy-ip-10-0-130-79.ec2.internal\" (UID: \"bc3d8b9722788fbfdcb8bc7f7b68e6ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.483731 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.483705    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f016832ad3a15792c6b1a23ee96cdaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"2f016832ad3a15792c6b1a23ee96cdaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.567168 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.567132    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 20:20:51.584520 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.584492    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f016832ad3a15792c6b1a23ee96cdaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"2f016832ad3a15792c6b1a23ee96cdaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.584592 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.584529    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f016832ad3a15792c6b1a23ee96cdaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"2f016832ad3a15792c6b1a23ee96cdaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.584592 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.584558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc3d8b9722788fbfdcb8bc7f7b68e6ec-config\") pod \"kube-apiserver-proxy-ip-10-0-130-79.ec2.internal\" (UID: \"bc3d8b9722788fbfdcb8bc7f7b68e6ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.584692 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.584589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f016832ad3a15792c6b1a23ee96cdaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"2f016832ad3a15792c6b1a23ee96cdaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.584692 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.584588    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc3d8b9722788fbfdcb8bc7f7b68e6ec-config\") pod \"kube-apiserver-proxy-ip-10-0-130-79.ec2.internal\" (UID: \"bc3d8b9722788fbfdcb8bc7f7b68e6ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.584692 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.584622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f016832ad3a15792c6b1a23ee96cdaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"2f016832ad3a15792c6b1a23ee96cdaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.667960 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.667885    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 20:20:51.743441 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.743413    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.748056 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:51.747822    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:51.768903 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.768873    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 20:20:51.869455 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.869408    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 20:20:51.970001 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:51.969907    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 20:20:52.070586 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:52.070542    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found"
Sep 18 20:20:52.102005 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.101981    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 20:20:52.102569 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.102113    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 20:20:52.102569 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.102154    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 20:20:52.154254 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.154230    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:52.181223 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.181202    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:52.181360 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.181250    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 20:20:52.183438 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.183413    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 20:15:51 +0000 UTC" deadline="2028-04-16 02:58:10.773938493 +0000 UTC"
Sep 18 20:20:52.183485 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.183438    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13806h37m18.590502989s"
Sep 18 20:20:52.192368 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.192350    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 20:20:52.195255 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.195239    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 20:20:52.196871 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.196851    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"
Sep 18 20:20:52.205205 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.205189    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 20:20:52.212378 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.212360    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bm2xf"
Sep 18 20:20:52.220536 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.220481    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bm2xf"
Sep 18 20:20:52.295191 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:52.295162    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f016832ad3a15792c6b1a23ee96cdaa.slice/crio-22ddececcebc1872d5ada8a0c89266f36bd6e32f90279b85dda6f76b3e3e4a58 WatchSource:0}: Error finding container 22ddececcebc1872d5ada8a0c89266f36bd6e32f90279b85dda6f76b3e3e4a58: Status 404 returned error can't find the container with id 22ddececcebc1872d5ada8a0c89266f36bd6e32f90279b85dda6f76b3e3e4a58
Sep 18 20:20:52.295679 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:52.295659    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3d8b9722788fbfdcb8bc7f7b68e6ec.slice/crio-be46a0c7aa480adc7a50d35c808626e2ceb909b30707ebf1e9ebd6387891319c WatchSource:0}: Error finding container be46a0c7aa480adc7a50d35c808626e2ceb909b30707ebf1e9ebd6387891319c: Status 404 returned error can't find the container with id be46a0c7aa480adc7a50d35c808626e2ceb909b30707ebf1e9ebd6387891319c
Sep 18 20:20:52.300073 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.300057    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:20:52.313296 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.313239    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" event={"ID":"2f016832ad3a15792c6b1a23ee96cdaa","Type":"ContainerStarted","Data":"22ddececcebc1872d5ada8a0c89266f36bd6e32f90279b85dda6f76b3e3e4a58"}
Sep 18 20:20:52.314486 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.314464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" event={"ID":"bc3d8b9722788fbfdcb8bc7f7b68e6ec","Type":"ContainerStarted","Data":"be46a0c7aa480adc7a50d35c808626e2ceb909b30707ebf1e9ebd6387891319c"}
Sep 18 20:20:52.666336 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:52.666261    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:53.009322 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.009290    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:53.131583 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.131546    2583 apiserver.go:52] "Watching apiserver"
Sep 18 20:20:53.141226 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.141191    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 20:20:53.141678 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.141652    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-98jdh","openshift-ovn-kubernetes/ovnkube-node-m4xrg","kube-system/konnectivity-agent-rlvbw","kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms","openshift-cluster-node-tuning-operator/tuned-nkg67","openshift-dns/node-resolver-jt29h","openshift-multus/multus-additional-cni-plugins-p55j2","openshift-multus/network-metrics-daemon-wkf7p","kube-system/global-pull-secret-syncer-9rj8d","openshift-image-registry/node-ca-dmm78","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal","openshift-multus/multus-frkc9","openshift-network-diagnostics/network-check-target-qvkkm"]
Sep 18 20:20:53.143336 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.143309    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:53.143444 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.143407    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:20:53.144368 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.144343    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.146000 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.145563    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rlvbw"
Sep 18 20:20:53.146867 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.146846    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:20:53.146954 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.146913    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:20:53.147920 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.147899    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 20:20:53.148116 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.148093    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 20:20:53.148242 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.148222    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zz9tb\""
Sep 18 20:20:53.148296 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.148253    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4v68c\""
Sep 18 20:20:53.148369 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.148352    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 20:20:53.148471 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.148457    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 20:20:53.148529 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.148495    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 20:20:53.148576 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.148528    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 20:20:53.148701 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.148683    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.149819 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.149802    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.151136 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.150987    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:20:53.151136 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.151064    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:20:53.151288 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.151191    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 20:20:53.151469 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.151450    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 20:20:53.151543 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.151478    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lqjpx\""
Sep 18 20:20:53.151614 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.151588    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 20:20:53.152589 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.152111    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 20:20:53.152589 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.152188    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jt29h"
Sep 18 20:20:53.152589 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.152464    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-blnlc\""
Sep 18 20:20:53.152733 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.152690    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 20:20:53.153737 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.153714    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.154903 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.154885    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 20:20:53.155400 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.155102    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 20:20:53.155400 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.155351    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ctj5q\""
Sep 18 20:20:53.155598 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.155514    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-98jdh"
Sep 18 20:20:53.156240 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.156223    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 20:20:53.156597 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.156580    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 20:20:53.156776 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.156756    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 20:20:53.157081 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.157066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dmm78"
Sep 18 20:20:53.157809 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.157791    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 20:20:53.157942 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.157925    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.158046 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.158019    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x4hql\""
Sep 18 20:20:53.158394 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.158376    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 20:20:53.158653 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.158637    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 20:20:53.158767 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.158754    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f4zkm\""
Sep 18 20:20:53.158984 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.158967    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 20:20:53.159067 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.159055    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 20:20:53.159291 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.159278    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 20:20:53.159474 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.159284    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 20:20:53.159509 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.159475    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b2pt7\""
Sep 18 20:20:53.160343 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.160328    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 20:20:53.160752 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.160337    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 20:20:53.161003 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.160986    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 20:20:53.161080 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.161017    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 20:20:53.162053 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.162032    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bdqq7\""
Sep 18 20:20:53.182941 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.182922    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 20:20:53.194329 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcvfb\" (UniqueName: \"kubernetes.io/projected/cf1f94f2-24f9-4064-a595-ae2716f9a047-kube-api-access-pcvfb\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.194329 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwq6l\" (UniqueName: \"kubernetes.io/projected/c011d236-b524-45ce-a873-88ca80a4c3d9-kube-api-access-rwq6l\") pod \"iptables-alerter-98jdh\" (UID: \"c011d236-b524-45ce-a873-88ca80a4c3d9\") " pod="openshift-network-operator/iptables-alerter-98jdh"
Sep 18 20:20:53.194480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-var-lib-cni-bin\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.194480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194369    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-etc-selinux\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.194480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194392    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bff1259-e240-4481-a8ea-a6ddc0e0ee6a-host\") pod \"node-ca-dmm78\" (UID: \"4bff1259-e240-4481-a8ea-a6ddc0e0ee6a\") " pod="openshift-image-registry/node-ca-dmm78"
Sep 18 20:20:53.194480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-sys\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.194480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl5f6\" (UniqueName: \"kubernetes.io/projected/eaf22891-e55a-433a-a689-fb5b3e3575f9-kube-api-access-jl5f6\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:20:53.194480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1003b80d-d1eb-4bf7-a93d-96fc66cdd16d-hosts-file\") pod \"node-resolver-jt29h\" (UID: \"1003b80d-d1eb-4bf7-a93d-96fc66cdd16d\") " pod="openshift-dns/node-resolver-jt29h"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/941899a6-9517-410e-b439-94c967f6c162-kubelet-config\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194505    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhg4g\" (UniqueName: \"kubernetes.io/projected/4b5e1706-b06b-4fd9-b342-9af08311c3af-kube-api-access-fhg4g\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-sysctl-d\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194557    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-slash\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-run-netns\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194628    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b5e1706-b06b-4fd9-b342-9af08311c3af-system-cni-dir\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-host\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f0229ef-b6b0-44da-beac-0973f4229671-tmp\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvnjz\" (UniqueName: \"kubernetes.io/projected/9f0229ef-b6b0-44da-beac-0973f4229671-kube-api-access-pvnjz\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcp5f\" (UniqueName: \"kubernetes.io/projected/1003b80d-d1eb-4bf7-a93d-96fc66cdd16d-kube-api-access-bcp5f\") pod \"node-resolver-jt29h\" (UID: \"1003b80d-d1eb-4bf7-a93d-96fc66cdd16d\") " pod="openshift-dns/node-resolver-jt29h"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194748    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-node-log\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194771    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4bff1259-e240-4481-a8ea-a6ddc0e0ee6a-serviceca\") pod \"node-ca-dmm78\" (UID: \"4bff1259-e240-4481-a8ea-a6ddc0e0ee6a\") " pod="openshift-image-registry/node-ca-dmm78"
Sep 18 20:20:53.194789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-sysctl-conf\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/059bd389-2748-46bb-bb0f-6b57cc6d391d-env-overrides\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf1f94f2-24f9-4064-a595-ae2716f9a047-cni-binary-copy\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-etc-openvswitch\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c011d236-b524-45ce-a873-88ca80a4c3d9-iptables-alerter-script\") pod \"iptables-alerter-98jdh\" (UID: \"c011d236-b524-45ce-a873-88ca80a4c3d9\") " pod="openshift-network-operator/iptables-alerter-98jdh"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194970    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-multus-conf-dir\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.194994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-modprobe-d\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195016    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-run\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195038    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-run-netns\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195060    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195110    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-var-lib-openvswitch\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195152    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b5e1706-b06b-4fd9-b342-9af08311c3af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-lib-modules\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195204    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjm6c\" (UniqueName: \"kubernetes.io/projected/4bff1259-e240-4481-a8ea-a6ddc0e0ee6a-kube-api-access-bjm6c\") pod \"node-ca-dmm78\" (UID: \"4bff1259-e240-4481-a8ea-a6ddc0e0ee6a\") " pod="openshift-image-registry/node-ca-dmm78"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1003b80d-d1eb-4bf7-a93d-96fc66cdd16d-tmp-dir\") pod \"node-resolver-jt29h\" (UID: \"1003b80d-d1eb-4bf7-a93d-96fc66cdd16d\") " pod="openshift-dns/node-resolver-jt29h"
Sep 18 20:20:53.195446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-cnibin\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195299    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-run-k8s-cni-cncf-io\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-run-ovn-kubernetes\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-cni-netd\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b5e1706-b06b-4fd9-b342-9af08311c3af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-sysconfig\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-etc-kubernetes\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195456    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195480    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-system-cni-dir\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-multus-socket-dir-parent\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195527    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-var-lib-kubelet\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-registration-dir\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195574    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-run-systemd\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195600    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/059bd389-2748-46bb-bb0f-6b57cc6d391d-ovnkube-script-lib\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-device-dir\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195648    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-log-socket\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.196228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195671    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86c8b3bd-a424-4004-9c8d-36ae6030370b-konnectivity-ca\") pod \"konnectivity-agent-rlvbw\" (UID: \"86c8b3bd-a424-4004-9c8d-36ae6030370b\") " pod="kube-system/konnectivity-agent-rlvbw"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pp4l\" (UniqueName: \"kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l\") pod \"network-check-target-qvkkm\" (UID: \"1ecf386e-6ab3-4448-9c76-75a96b3cfa75\") " pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-systemd\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195747    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf1f94f2-24f9-4064-a595-ae2716f9a047-multus-daemon-config\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-multus-cni-dir\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c011d236-b524-45ce-a873-88ca80a4c3d9-host-slash\") pod \"iptables-alerter-98jdh\" (UID: \"c011d236-b524-45ce-a873-88ca80a4c3d9\") " pod="openshift-network-operator/iptables-alerter-98jdh"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195829    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-var-lib-cni-multus\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-socket-dir\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-kubelet\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-cni-bin\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/059bd389-2748-46bb-bb0f-6b57cc6d391d-ovnkube-config\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-hostroot\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.195980    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-systemd-units\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b5e1706-b06b-4fd9-b342-9af08311c3af-cnibin\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196030    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b5e1706-b06b-4fd9-b342-9af08311c3af-cni-binary-copy\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196059    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b5e1706-b06b-4fd9-b342-9af08311c3af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.196964 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86c8b3bd-a424-4004-9c8d-36ae6030370b-agent-certs\") pod \"konnectivity-agent-rlvbw\" (UID: \"86c8b3bd-a424-4004-9c8d-36ae6030370b\") " pod="kube-system/konnectivity-agent-rlvbw"
Sep 18 20:20:53.197545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-kubernetes\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.197545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196158    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-var-lib-kubelet\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.197545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196185    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/941899a6-9517-410e-b439-94c967f6c162-dbus\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:53.197545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196213    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-run-multus-certs\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.197545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-sys-fs\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.197545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196450    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-run-openvswitch\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.197545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/059bd389-2748-46bb-bb0f-6b57cc6d391d-ovn-node-metrics-cert\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.197545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196516    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk8fc\" (UniqueName: \"kubernetes.io/projected/059bd389-2748-46bb-bb0f-6b57cc6d391d-kube-api-access-hk8fc\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.197545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196544    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fqf\" (UniqueName: \"kubernetes.io/projected/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-kube-api-access-q7fqf\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.197545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-run-ovn\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.197545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.196610    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f0229ef-b6b0-44da-beac-0973f4229671-etc-tuned\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.222187 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.222140    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 20:15:52 +0000 UTC" deadline="2028-04-06 00:35:47.545860174 +0000 UTC"
Sep 18 20:20:53.222292 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.222189    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13564h14m54.323675338s"
Sep 18 20:20:53.297484 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcvfb\" (UniqueName: \"kubernetes.io/projected/cf1f94f2-24f9-4064-a595-ae2716f9a047-kube-api-access-pcvfb\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.297659 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwq6l\" (UniqueName: \"kubernetes.io/projected/c011d236-b524-45ce-a873-88ca80a4c3d9-kube-api-access-rwq6l\") pod \"iptables-alerter-98jdh\" (UID: \"c011d236-b524-45ce-a873-88ca80a4c3d9\") " pod="openshift-network-operator/iptables-alerter-98jdh"
Sep 18 20:20:53.297659 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-var-lib-cni-bin\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.297659 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-etc-selinux\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.297659 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bff1259-e240-4481-a8ea-a6ddc0e0ee6a-host\") pod \"node-ca-dmm78\" (UID: \"4bff1259-e240-4481-a8ea-a6ddc0e0ee6a\") " pod="openshift-image-registry/node-ca-dmm78"
Sep 18 20:20:53.297659 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-sys\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.297659 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5f6\" (UniqueName: \"kubernetes.io/projected/eaf22891-e55a-433a-a689-fb5b3e3575f9-kube-api-access-jl5f6\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:20:53.297659 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1003b80d-d1eb-4bf7-a93d-96fc66cdd16d-hosts-file\") pod \"node-resolver-jt29h\" (UID: \"1003b80d-d1eb-4bf7-a93d-96fc66cdd16d\") " pod="openshift-dns/node-resolver-jt29h"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297677    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/941899a6-9517-410e-b439-94c967f6c162-kubelet-config\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhg4g\" (UniqueName: \"kubernetes.io/projected/4b5e1706-b06b-4fd9-b342-9af08311c3af-kube-api-access-fhg4g\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297731    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-sys\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-var-lib-cni-bin\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-sysctl-d\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/941899a6-9517-410e-b439-94c967f6c162-kubelet-config\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-slash\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297833    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-slash\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-run-netns\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297868    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-sysctl-d\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b5e1706-b06b-4fd9-b342-9af08311c3af-system-cni-dir\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297900    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-host\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.297909    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f0229ef-b6b0-44da-beac-0973f4229671-tmp\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1003b80d-d1eb-4bf7-a93d-96fc66cdd16d-hosts-file\") pod \"node-resolver-jt29h\" (UID: \"1003b80d-d1eb-4bf7-a93d-96fc66cdd16d\") " pod="openshift-dns/node-resolver-jt29h"
Sep 18 20:20:53.298066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297785    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bff1259-e240-4481-a8ea-a6ddc0e0ee6a-host\") pod \"node-ca-dmm78\" (UID: \"4bff1259-e240-4481-a8ea-a6ddc0e0ee6a\") " pod="openshift-image-registry/node-ca-dmm78"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.297982    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs podName:eaf22891-e55a-433a-a689-fb5b3e3575f9 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:53.797943388 +0000 UTC m=+4.705845648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs") pod "network-metrics-daemon-wkf7p" (UID: "eaf22891-e55a-433a-a689-fb5b3e3575f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297719    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-etc-selinux\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-run-netns\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvnjz\" (UniqueName: \"kubernetes.io/projected/9f0229ef-b6b0-44da-beac-0973f4229671-kube-api-access-pvnjz\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcp5f\" (UniqueName: \"kubernetes.io/projected/1003b80d-d1eb-4bf7-a93d-96fc66cdd16d-kube-api-access-bcp5f\") pod \"node-resolver-jt29h\" (UID: \"1003b80d-d1eb-4bf7-a93d-96fc66cdd16d\") " pod="openshift-dns/node-resolver-jt29h"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-node-log\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4bff1259-e240-4481-a8ea-a6ddc0e0ee6a-serviceca\") pod \"node-ca-dmm78\" (UID: \"4bff1259-e240-4481-a8ea-a6ddc0e0ee6a\") " pod="openshift-image-registry/node-ca-dmm78"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.297899    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b5e1706-b06b-4fd9-b342-9af08311c3af-system-cni-dir\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-sysctl-conf\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298102    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/059bd389-2748-46bb-bb0f-6b57cc6d391d-env-overrides\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf1f94f2-24f9-4064-a595-ae2716f9a047-cni-binary-copy\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-host\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-etc-openvswitch\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298220    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-node-log\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c011d236-b524-45ce-a873-88ca80a4c3d9-iptables-alerter-script\") pod \"iptables-alerter-98jdh\" (UID: \"c011d236-b524-45ce-a873-88ca80a4c3d9\") " pod="openshift-network-operator/iptables-alerter-98jdh"
Sep 18 20:20:53.298882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298299    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-multus-conf-dir\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-modprobe-d\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-run\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-run-netns\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298387    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-var-lib-openvswitch\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b5e1706-b06b-4fd9-b342-9af08311c3af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-lib-modules\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjm6c\" (UniqueName: \"kubernetes.io/projected/4bff1259-e240-4481-a8ea-a6ddc0e0ee6a-kube-api-access-bjm6c\") pod \"node-ca-dmm78\" (UID: \"4bff1259-e240-4481-a8ea-a6ddc0e0ee6a\") " pod="openshift-image-registry/node-ca-dmm78"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1003b80d-d1eb-4bf7-a93d-96fc66cdd16d-tmp-dir\") pod \"node-resolver-jt29h\" (UID: \"1003b80d-d1eb-4bf7-a93d-96fc66cdd16d\") " pod="openshift-dns/node-resolver-jt29h"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-cnibin\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298568    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-etc-openvswitch\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-run-k8s-cni-cncf-io\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298604    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-run-ovn-kubernetes\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-cni-netd\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.299720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b5e1706-b06b-4fd9-b342-9af08311c3af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298673    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-sysconfig\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298689    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-sysctl-conf\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298695    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-etc-kubernetes\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298708    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf1f94f2-24f9-4064-a595-ae2716f9a047-cni-binary-copy\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-system-cni-dir\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298781    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-multus-socket-dir-parent\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-system-cni-dir\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-var-lib-kubelet\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-registration-dir\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298845    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-run-systemd\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/059bd389-2748-46bb-bb0f-6b57cc6d391d-ovnkube-script-lib\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-device-dir\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298917    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-log-socket\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86c8b3bd-a424-4004-9c8d-36ae6030370b-konnectivity-ca\") pod \"konnectivity-agent-rlvbw\" (UID: \"86c8b3bd-a424-4004-9c8d-36ae6030370b\") " pod="kube-system/konnectivity-agent-rlvbw"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pp4l\" (UniqueName: \"kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l\") pod \"network-check-target-qvkkm\" (UID: \"1ecf386e-6ab3-4448-9c76-75a96b3cfa75\") " pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:20:53.300423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.298983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-systemd\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299004    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf1f94f2-24f9-4064-a595-ae2716f9a047-multus-daemon-config\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-multus-cni-dir\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c011d236-b524-45ce-a873-88ca80a4c3d9-host-slash\") pod \"iptables-alerter-98jdh\" (UID: \"c011d236-b524-45ce-a873-88ca80a4c3d9\") " pod="openshift-network-operator/iptables-alerter-98jdh"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-var-lib-cni-multus\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-socket-dir\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-kubelet\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-cni-bin\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/059bd389-2748-46bb-bb0f-6b57cc6d391d-env-overrides\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299240    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-var-lib-kubelet\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c011d236-b524-45ce-a873-88ca80a4c3d9-iptables-alerter-script\") pod \"iptables-alerter-98jdh\" (UID: \"c011d236-b524-45ce-a873-88ca80a4c3d9\") " pod="openshift-network-operator/iptables-alerter-98jdh"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/059bd389-2748-46bb-bb0f-6b57cc6d391d-ovnkube-config\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-hostroot\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299410    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-multus-conf-dir\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-systemd-units\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b5e1706-b06b-4fd9-b342-9af08311c3af-cnibin\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299457    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-registration-dir\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.301162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b5e1706-b06b-4fd9-b342-9af08311c3af-cni-binary-copy\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b5e1706-b06b-4fd9-b342-9af08311c3af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299495    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-run-systemd\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299543    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299584    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-run\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-run-netns\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/059bd389-2748-46bb-bb0f-6b57cc6d391d-ovnkube-config\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301170    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-multus-cni-dir\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301233    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-cni-netd\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86c8b3bd-a424-4004-9c8d-36ae6030370b-agent-certs\") pod \"konnectivity-agent-rlvbw\" (UID: \"86c8b3bd-a424-4004-9c8d-36ae6030370b\") " pod="kube-system/konnectivity-agent-rlvbw"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-kubernetes\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4bff1259-e240-4481-a8ea-a6ddc0e0ee6a-serviceca\") pod \"node-ca-dmm78\" (UID: \"4bff1259-e240-4481-a8ea-a6ddc0e0ee6a\") " pod="openshift-image-registry/node-ca-dmm78"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-var-lib-kubelet\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301684    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/941899a6-9517-410e-b439-94c967f6c162-dbus\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-log-socket\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-run-multus-certs\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-sys-fs\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-run-openvswitch\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.301925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f0229ef-b6b0-44da-beac-0973f4229671-tmp\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/059bd389-2748-46bb-bb0f-6b57cc6d391d-ovn-node-metrics-cert\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b5e1706-b06b-4fd9-b342-9af08311c3af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk8fc\" (UniqueName: \"kubernetes.io/projected/059bd389-2748-46bb-bb0f-6b57cc6d391d-kube-api-access-hk8fc\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fqf\" (UniqueName: \"kubernetes.io/projected/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-kube-api-access-q7fqf\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/059bd389-2748-46bb-bb0f-6b57cc6d391d-ovnkube-script-lib\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301915    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-etc-kubernetes\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-run-ovn\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.301955    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f0229ef-b6b0-44da-beac-0973f4229671-etc-tuned\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.301998    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.302060    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret podName:941899a6-9517-410e-b439-94c967f6c162 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:53.802042213 +0000 UTC m=+4.709944469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret") pod "global-pull-secret-syncer-9rj8d" (UID: "941899a6-9517-410e-b439-94c967f6c162") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302080    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-kubernetes\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302157    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86c8b3bd-a424-4004-9c8d-36ae6030370b-konnectivity-ca\") pod \"konnectivity-agent-rlvbw\" (UID: \"86c8b3bd-a424-4004-9c8d-36ae6030370b\") " pod="kube-system/konnectivity-agent-rlvbw"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302176    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-var-lib-kubelet\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302237    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-device-dir\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302372    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/941899a6-9517-410e-b439-94c967f6c162-dbus\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302436    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-systemd\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.302746 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302453    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-run-multus-certs\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302476    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-hostroot\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302535    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1003b80d-d1eb-4bf7-a93d-96fc66cdd16d-tmp-dir\") pod \"node-resolver-jt29h\" (UID: \"1003b80d-d1eb-4bf7-a93d-96fc66cdd16d\") " pod="openshift-dns/node-resolver-jt29h"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302547    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-systemd-units\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302549    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-sys-fs\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302589    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b5e1706-b06b-4fd9-b342-9af08311c3af-cnibin\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302641    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-run-k8s-cni-cncf-io\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302659    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c011d236-b524-45ce-a873-88ca80a4c3d9-host-slash\") pod \"iptables-alerter-98jdh\" (UID: \"c011d236-b524-45ce-a873-88ca80a4c3d9\") " pod="openshift-network-operator/iptables-alerter-98jdh"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-run-ovn-kubernetes\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302741    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-host-var-lib-cni-multus\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.302834    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-lib-modules\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.303169    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-cnibin\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.303215    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b5e1706-b06b-4fd9-b342-9af08311c3af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.303247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-kubelet\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.303271    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-sysconfig\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.303299    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-run-ovn\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.299499    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9f0229ef-b6b0-44da-beac-0973f4229671-etc-modprobe-d\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.303325    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b5e1706-b06b-4fd9-b342-9af08311c3af-cni-binary-copy\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.303538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.303426    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-socket-dir\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.304349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.303325    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-host-cni-bin\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.304349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.303465    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b5e1706-b06b-4fd9-b342-9af08311c3af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.304349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.303550    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-run-openvswitch\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.304349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.303594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/059bd389-2748-46bb-bb0f-6b57cc6d391d-var-lib-openvswitch\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.304349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.303809    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf1f94f2-24f9-4064-a595-ae2716f9a047-multus-daemon-config\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.305510 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.305485    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf1f94f2-24f9-4064-a595-ae2716f9a047-multus-socket-dir-parent\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.305610 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.305548    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9f0229ef-b6b0-44da-beac-0973f4229671-etc-tuned\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.307478 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.307452    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/059bd389-2748-46bb-bb0f-6b57cc6d391d-ovn-node-metrics-cert\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.308084 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.308060    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:53.308190 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.308092    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:53.308190 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.308138    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6pp4l for pod openshift-network-diagnostics/network-check-target-qvkkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:53.308308 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.308208    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l podName:1ecf386e-6ab3-4448-9c76-75a96b3cfa75 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:53.808190187 +0000 UTC m=+4.716092465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6pp4l" (UniqueName: "kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l") pod "network-check-target-qvkkm" (UID: "1ecf386e-6ab3-4448-9c76-75a96b3cfa75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:53.309963 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.309943    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhg4g\" (UniqueName: \"kubernetes.io/projected/4b5e1706-b06b-4fd9-b342-9af08311c3af-kube-api-access-fhg4g\") pod \"multus-additional-cni-plugins-p55j2\" (UID: \"4b5e1706-b06b-4fd9-b342-9af08311c3af\") " pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.310844 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.310644    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcvfb\" (UniqueName: \"kubernetes.io/projected/cf1f94f2-24f9-4064-a595-ae2716f9a047-kube-api-access-pcvfb\") pod \"multus-frkc9\" (UID: \"cf1f94f2-24f9-4064-a595-ae2716f9a047\") " pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.311086 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.311061    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwq6l\" (UniqueName: \"kubernetes.io/projected/c011d236-b524-45ce-a873-88ca80a4c3d9-kube-api-access-rwq6l\") pod \"iptables-alerter-98jdh\" (UID: \"c011d236-b524-45ce-a873-88ca80a4c3d9\") " pod="openshift-network-operator/iptables-alerter-98jdh"
Sep 18 20:20:53.312325 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.312292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjm6c\" (UniqueName: \"kubernetes.io/projected/4bff1259-e240-4481-a8ea-a6ddc0e0ee6a-kube-api-access-bjm6c\") pod \"node-ca-dmm78\" (UID: \"4bff1259-e240-4481-a8ea-a6ddc0e0ee6a\") " pod="openshift-image-registry/node-ca-dmm78"
Sep 18 20:20:53.312325 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.312299    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcp5f\" (UniqueName: \"kubernetes.io/projected/1003b80d-d1eb-4bf7-a93d-96fc66cdd16d-kube-api-access-bcp5f\") pod \"node-resolver-jt29h\" (UID: \"1003b80d-d1eb-4bf7-a93d-96fc66cdd16d\") " pod="openshift-dns/node-resolver-jt29h"
Sep 18 20:20:53.312685 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.312659    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86c8b3bd-a424-4004-9c8d-36ae6030370b-agent-certs\") pod \"konnectivity-agent-rlvbw\" (UID: \"86c8b3bd-a424-4004-9c8d-36ae6030370b\") " pod="kube-system/konnectivity-agent-rlvbw"
Sep 18 20:20:53.312860 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.312839    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl5f6\" (UniqueName: \"kubernetes.io/projected/eaf22891-e55a-433a-a689-fb5b3e3575f9-kube-api-access-jl5f6\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:20:53.314236 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.314214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fqf\" (UniqueName: \"kubernetes.io/projected/e55e670d-adf8-4a8d-8c6c-bf8956b86ae2-kube-api-access-q7fqf\") pod \"aws-ebs-csi-driver-node-spsms\" (UID: \"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.314381 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.314350    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvnjz\" (UniqueName: \"kubernetes.io/projected/9f0229ef-b6b0-44da-beac-0973f4229671-kube-api-access-pvnjz\") pod \"tuned-nkg67\" (UID: \"9f0229ef-b6b0-44da-beac-0973f4229671\") " pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.314992 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.314959    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk8fc\" (UniqueName: \"kubernetes.io/projected/059bd389-2748-46bb-bb0f-6b57cc6d391d-kube-api-access-hk8fc\") pod \"ovnkube-node-m4xrg\" (UID: \"059bd389-2748-46bb-bb0f-6b57cc6d391d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.461757 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.461668    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-frkc9"
Sep 18 20:20:53.471740 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.471718    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rlvbw"
Sep 18 20:20:53.479389 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.479370    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms"
Sep 18 20:20:53.485536 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.485518    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nkg67"
Sep 18 20:20:53.492074 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.492049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jt29h"
Sep 18 20:20:53.502392 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.502372    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:20:53.509911 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.509893    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-98jdh"
Sep 18 20:20:53.516390 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.516373    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dmm78"
Sep 18 20:20:53.522028 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.522010    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p55j2"
Sep 18 20:20:53.577539 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.577509    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:53.806374 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.806301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:53.806374 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.806355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:20:53.806554 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.806447    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:53.806554 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.806450    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:53.806554 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.806499    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs podName:eaf22891-e55a-433a-a689-fb5b3e3575f9 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:54.806481874 +0000 UTC m=+5.714384131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs") pod "network-metrics-daemon-wkf7p" (UID: "eaf22891-e55a-433a-a689-fb5b3e3575f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:53.806554 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.806532    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret podName:941899a6-9517-410e-b439-94c967f6c162 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:54.806518271 +0000 UTC m=+5.714420528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret") pod "global-pull-secret-syncer-9rj8d" (UID: "941899a6-9517-410e-b439-94c967f6c162") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:53.907422 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:53.907386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pp4l\" (UniqueName: \"kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l\") pod \"network-check-target-qvkkm\" (UID: \"1ecf386e-6ab3-4448-9c76-75a96b3cfa75\") " pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:20:53.907583 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.907565    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:53.907631 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.907594    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:53.907631 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.907608    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6pp4l for pod openshift-network-diagnostics/network-check-target-qvkkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:53.907731 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:53.907673    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l podName:1ecf386e-6ab3-4448-9c76-75a96b3cfa75 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:54.907652309 +0000 UTC m=+5.815554610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6pp4l" (UniqueName: "kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l") pod "network-check-target-qvkkm" (UID: "1ecf386e-6ab3-4448-9c76-75a96b3cfa75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:54.148248 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:54.148224    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc011d236_b524_45ce_a873_88ca80a4c3d9.slice/crio-e97591b37b23ed17a1fb11e928da84d0464d32030dba3b4a5f9e8e45829d7e25 WatchSource:0}: Error finding container e97591b37b23ed17a1fb11e928da84d0464d32030dba3b4a5f9e8e45829d7e25: Status 404 returned error can't find the container with id e97591b37b23ed17a1fb11e928da84d0464d32030dba3b4a5f9e8e45829d7e25
Sep 18 20:20:54.149483 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:54.149459    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059bd389_2748_46bb_bb0f_6b57cc6d391d.slice/crio-7a8a2297309626de2b50ef47c80c827453830f9887f3c98714927018002ecd2b WatchSource:0}: Error finding container 7a8a2297309626de2b50ef47c80c827453830f9887f3c98714927018002ecd2b: Status 404 returned error can't find the container with id 7a8a2297309626de2b50ef47c80c827453830f9887f3c98714927018002ecd2b
Sep 18 20:20:54.151949 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:54.150433    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bff1259_e240_4481_a8ea_a6ddc0e0ee6a.slice/crio-c45d6df577cbabff19559d9bc74a972fc47c6ee33e9a70a2ed3febd52deb366f WatchSource:0}: Error finding container c45d6df577cbabff19559d9bc74a972fc47c6ee33e9a70a2ed3febd52deb366f: Status 404 returned error can't find the container with id c45d6df577cbabff19559d9bc74a972fc47c6ee33e9a70a2ed3febd52deb366f
Sep 18 20:20:54.154607 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:54.154584    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5e1706_b06b_4fd9_b342_9af08311c3af.slice/crio-c98ab22ae8ac2439f6e87408a5f4bfb3f7ad965fae47bcf7c69e91d256b01ec5 WatchSource:0}: Error finding container c98ab22ae8ac2439f6e87408a5f4bfb3f7ad965fae47bcf7c69e91d256b01ec5: Status 404 returned error can't find the container with id c98ab22ae8ac2439f6e87408a5f4bfb3f7ad965fae47bcf7c69e91d256b01ec5
Sep 18 20:20:54.156243 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:54.156223    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55e670d_adf8_4a8d_8c6c_bf8956b86ae2.slice/crio-ae8a6e6668ac9eb2b3e9f9eca71852e8a3deee9cf492e9a7f559870bacd7e32f WatchSource:0}: Error finding container ae8a6e6668ac9eb2b3e9f9eca71852e8a3deee9cf492e9a7f559870bacd7e32f: Status 404 returned error can't find the container with id ae8a6e6668ac9eb2b3e9f9eca71852e8a3deee9cf492e9a7f559870bacd7e32f
Sep 18 20:20:54.158173 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:54.158065    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c8b3bd_a424_4004_9c8d_36ae6030370b.slice/crio-0b9e4cf81006f27166b931f30336934eba583878e8d73d21df21a3420669063c WatchSource:0}: Error finding container 0b9e4cf81006f27166b931f30336934eba583878e8d73d21df21a3420669063c: Status 404 returned error can't find the container with id 0b9e4cf81006f27166b931f30336934eba583878e8d73d21df21a3420669063c
Sep 18 20:20:54.159152 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:54.158689    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0229ef_b6b0_44da_beac_0973f4229671.slice/crio-6adba4c4bfb5293250254a2c2e52a066b32b919c670ca6793dec752d0ae04880 WatchSource:0}: Error finding container 6adba4c4bfb5293250254a2c2e52a066b32b919c670ca6793dec752d0ae04880: Status 404 returned error can't find the container with id 6adba4c4bfb5293250254a2c2e52a066b32b919c670ca6793dec752d0ae04880
Sep 18 20:20:54.159557 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:20:54.159527    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1003b80d_d1eb_4bf7_a93d_96fc66cdd16d.slice/crio-1ed225288127f57ae46f57fc84acf953a66891504857b2efd2305c4884438c3b WatchSource:0}: Error finding container 1ed225288127f57ae46f57fc84acf953a66891504857b2efd2305c4884438c3b: Status 404 returned error can't find the container with id 1ed225288127f57ae46f57fc84acf953a66891504857b2efd2305c4884438c3b
Sep 18 20:20:54.222835 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.222683    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 20:15:52 +0000 UTC" deadline="2028-06-25 05:58:35.860583956 +0000 UTC"
Sep 18 20:20:54.222835 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.222835    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15489h37m41.637752331s"
Sep 18 20:20:54.310701 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.310674    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:54.310832 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:54.310777    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:20:54.310876 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.310826    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:20:54.310965 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:54.310908    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:20:54.318210 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.318182    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-98jdh" event={"ID":"c011d236-b524-45ce-a873-88ca80a4c3d9","Type":"ContainerStarted","Data":"e97591b37b23ed17a1fb11e928da84d0464d32030dba3b4a5f9e8e45829d7e25"}
Sep 18 20:20:54.320057 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.320039    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" event={"ID":"bc3d8b9722788fbfdcb8bc7f7b68e6ec","Type":"ContainerStarted","Data":"638dc1e38cfe2c02e8778cda17e86e205df40940595c7bd076b46e2b4fcb58c3"}
Sep 18 20:20:54.321304 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.321285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-frkc9" event={"ID":"cf1f94f2-24f9-4064-a595-ae2716f9a047","Type":"ContainerStarted","Data":"e9de94a7e48c63126c3bfeec6591a151899a95f5f32adcce9606250d5ee6b177"}
Sep 18 20:20:54.322452 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.322429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jt29h" event={"ID":"1003b80d-d1eb-4bf7-a93d-96fc66cdd16d","Type":"ContainerStarted","Data":"1ed225288127f57ae46f57fc84acf953a66891504857b2efd2305c4884438c3b"}
Sep 18 20:20:54.323617 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.323591    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nkg67" event={"ID":"9f0229ef-b6b0-44da-beac-0973f4229671","Type":"ContainerStarted","Data":"6adba4c4bfb5293250254a2c2e52a066b32b919c670ca6793dec752d0ae04880"}
Sep 18 20:20:54.324762 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.324742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms" event={"ID":"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2","Type":"ContainerStarted","Data":"ae8a6e6668ac9eb2b3e9f9eca71852e8a3deee9cf492e9a7f559870bacd7e32f"}
Sep 18 20:20:54.326692 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.326671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p55j2" event={"ID":"4b5e1706-b06b-4fd9-b342-9af08311c3af","Type":"ContainerStarted","Data":"c98ab22ae8ac2439f6e87408a5f4bfb3f7ad965fae47bcf7c69e91d256b01ec5"}
Sep 18 20:20:54.327930 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.327899    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dmm78" event={"ID":"4bff1259-e240-4481-a8ea-a6ddc0e0ee6a","Type":"ContainerStarted","Data":"c45d6df577cbabff19559d9bc74a972fc47c6ee33e9a70a2ed3febd52deb366f"}
Sep 18 20:20:54.329346 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.329320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rlvbw" event={"ID":"86c8b3bd-a424-4004-9c8d-36ae6030370b","Type":"ContainerStarted","Data":"0b9e4cf81006f27166b931f30336934eba583878e8d73d21df21a3420669063c"}
Sep 18 20:20:54.331850 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.331829    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg" event={"ID":"059bd389-2748-46bb-bb0f-6b57cc6d391d","Type":"ContainerStarted","Data":"7a8a2297309626de2b50ef47c80c827453830f9887f3c98714927018002ecd2b"}
Sep 18 20:20:54.336024 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.335985    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" podStartSLOduration=2.335973836 podStartE2EDuration="2.335973836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:54.33570304 +0000 UTC m=+5.243605321" watchObservedRunningTime="2026-09-18 20:20:54.335973836 +0000 UTC m=+5.243876114"
Sep 18 20:20:54.813699 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.813604    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:54.813699 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.813695    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:20:54.813918 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:54.813851    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:54.813968 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:54.813923    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs podName:eaf22891-e55a-433a-a689-fb5b3e3575f9 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:56.813904065 +0000 UTC m=+7.721806346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs") pod "network-metrics-daemon-wkf7p" (UID: "eaf22891-e55a-433a-a689-fb5b3e3575f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:54.814330 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:54.814309    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:54.814427 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:54.814366    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret podName:941899a6-9517-410e-b439-94c967f6c162 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:56.814352245 +0000 UTC m=+7.722254505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret") pod "global-pull-secret-syncer-9rj8d" (UID: "941899a6-9517-410e-b439-94c967f6c162") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:54.915062 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:54.915023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pp4l\" (UniqueName: \"kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l\") pod \"network-check-target-qvkkm\" (UID: \"1ecf386e-6ab3-4448-9c76-75a96b3cfa75\") " pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:20:54.915281 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:54.915242    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:54.915281 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:54.915270    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:54.915281 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:54.915283    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6pp4l for pod openshift-network-diagnostics/network-check-target-qvkkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:54.915476 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:54.915340    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l podName:1ecf386e-6ab3-4448-9c76-75a96b3cfa75 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:56.915322664 +0000 UTC m=+7.823224923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6pp4l" (UniqueName: "kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l") pod "network-check-target-qvkkm" (UID: "1ecf386e-6ab3-4448-9c76-75a96b3cfa75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:55.312650 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:55.312564    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:20:55.313074 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:55.312726    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:20:55.347478 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:55.346728    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f016832ad3a15792c6b1a23ee96cdaa" containerID="1105e01e47afae539eb39310f23e3f86f828203af03af1f35d5d0660529ab933" exitCode=0
Sep 18 20:20:55.347478 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:55.346882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" event={"ID":"2f016832ad3a15792c6b1a23ee96cdaa","Type":"ContainerDied","Data":"1105e01e47afae539eb39310f23e3f86f828203af03af1f35d5d0660529ab933"}
Sep 18 20:20:56.311080 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:56.310452    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:56.311080 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:56.310579    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:20:56.311080 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:56.310938    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:20:56.311080 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:56.311036    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:20:56.364386 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:56.364353    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" event={"ID":"2f016832ad3a15792c6b1a23ee96cdaa","Type":"ContainerStarted","Data":"d98befff5964093643bcb02efa46fbe08bf608aee510e9f6772fbfebd504b881"}
Sep 18 20:20:56.382250 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:56.382196    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" podStartSLOduration=4.382177025 podStartE2EDuration="4.382177025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:56.381504564 +0000 UTC m=+7.289406868" watchObservedRunningTime="2026-09-18 20:20:56.382177025 +0000 UTC m=+7.290079296"
Sep 18 20:20:56.830946 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:56.830906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:20:56.831154 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:56.830983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:56.831222 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:56.831153    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:56.831222 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:56.831216    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret podName:941899a6-9517-410e-b439-94c967f6c162 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:00.831197791 +0000 UTC m=+11.739100053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret") pod "global-pull-secret-syncer-9rj8d" (UID: "941899a6-9517-410e-b439-94c967f6c162") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:56.831651 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:56.831625    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:56.831721 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:56.831686    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs podName:eaf22891-e55a-433a-a689-fb5b3e3575f9 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:00.831668622 +0000 UTC m=+11.739570881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs") pod "network-metrics-daemon-wkf7p" (UID: "eaf22891-e55a-433a-a689-fb5b3e3575f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:56.931818 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:56.931755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pp4l\" (UniqueName: \"kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l\") pod \"network-check-target-qvkkm\" (UID: \"1ecf386e-6ab3-4448-9c76-75a96b3cfa75\") " pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:20:56.931990 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:56.931923    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:56.931990 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:56.931952    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:56.931990 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:56.931965    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6pp4l for pod openshift-network-diagnostics/network-check-target-qvkkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:56.932180 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:56.932084    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l podName:1ecf386e-6ab3-4448-9c76-75a96b3cfa75 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:00.932046556 +0000 UTC m=+11.839948815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6pp4l" (UniqueName: "kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l") pod "network-check-target-qvkkm" (UID: "1ecf386e-6ab3-4448-9c76-75a96b3cfa75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:57.316429 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:57.313754    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:20:57.316429 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:57.313933    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:20:58.310038 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:58.310002    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:20:58.310462 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:58.310175    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:20:58.310504 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:58.310494    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:20:58.310574 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:58.310556    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:20:59.311149 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:20:59.310862    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:20:59.311149 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:20:59.311014    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:00.310009 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:00.309969    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:00.310219 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:00.310137    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:00.310564 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:00.310544    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:00.310664 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:00.310643    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:00.865657 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:00.865360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:00.865657 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:00.865473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:00.865657 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:00.865509    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:21:00.865657 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:00.865573    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:21:00.865657 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:00.865579    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret podName:941899a6-9517-410e-b439-94c967f6c162 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:08.865558274 +0000 UTC m=+19.773460534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret") pod "global-pull-secret-syncer-9rj8d" (UID: "941899a6-9517-410e-b439-94c967f6c162") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:21:00.865657 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:00.865634    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs podName:eaf22891-e55a-433a-a689-fb5b3e3575f9 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:08.865617905 +0000 UTC m=+19.773520165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs") pod "network-metrics-daemon-wkf7p" (UID: "eaf22891-e55a-433a-a689-fb5b3e3575f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:21:00.966565 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:00.966529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pp4l\" (UniqueName: \"kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l\") pod \"network-check-target-qvkkm\" (UID: \"1ecf386e-6ab3-4448-9c76-75a96b3cfa75\") " pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:00.966741 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:00.966720    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:21:00.966815 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:00.966747    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:21:00.966815 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:00.966758    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6pp4l for pod openshift-network-diagnostics/network-check-target-qvkkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:21:00.966815 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:00.966812    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l podName:1ecf386e-6ab3-4448-9c76-75a96b3cfa75 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:08.966794445 +0000 UTC m=+19.874696707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6pp4l" (UniqueName: "kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l") pod "network-check-target-qvkkm" (UID: "1ecf386e-6ab3-4448-9c76-75a96b3cfa75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:21:01.311494 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:01.311417    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:01.311626 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:01.311516    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:02.309932 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:02.309896    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:02.310468 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:02.310006    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:02.310468 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:02.310380    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:02.310468 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:02.310456    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:03.310619 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:03.310585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:03.311038 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:03.310721    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:04.310627 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:04.310588    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:04.311051 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:04.310588    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:04.311051 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:04.310695    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:04.311051 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:04.310817    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:05.310700 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:05.310667    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:05.311196 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:05.310808    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:06.310473 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:06.310429    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:06.310682 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:06.310433    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:06.310682 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:06.310548    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:06.310682 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:06.310631    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:07.310513 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:07.310473    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:07.310686 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:07.310631    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:08.310773 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:08.310741    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:08.311193 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:08.310741    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:08.311193 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:08.310845    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:08.311193 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:08.310924    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:08.926143 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:08.926096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:08.926324 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:08.926203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:08.926324 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:08.926240    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:21:08.926324 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:08.926307    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret podName:941899a6-9517-410e-b439-94c967f6c162 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:24.926288279 +0000 UTC m=+35.834190537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret") pod "global-pull-secret-syncer-9rj8d" (UID: "941899a6-9517-410e-b439-94c967f6c162") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:21:08.926324 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:08.926311    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:21:08.926491 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:08.926377    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs podName:eaf22891-e55a-433a-a689-fb5b3e3575f9 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:24.926357738 +0000 UTC m=+35.834260000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs") pod "network-metrics-daemon-wkf7p" (UID: "eaf22891-e55a-433a-a689-fb5b3e3575f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:21:09.027306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:09.027269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pp4l\" (UniqueName: \"kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l\") pod \"network-check-target-qvkkm\" (UID: \"1ecf386e-6ab3-4448-9c76-75a96b3cfa75\") " pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:09.027505 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:09.027449    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:21:09.027505 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:09.027469    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:21:09.027505 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:09.027483    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6pp4l for pod openshift-network-diagnostics/network-check-target-qvkkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:21:09.027679 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:09.027535    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l podName:1ecf386e-6ab3-4448-9c76-75a96b3cfa75 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:25.027521887 +0000 UTC m=+35.935424145 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6pp4l" (UniqueName: "kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l") pod "network-check-target-qvkkm" (UID: "1ecf386e-6ab3-4448-9c76-75a96b3cfa75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:21:09.309866 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:09.309785    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:09.310030 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:09.309911    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:10.310117 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:10.309855    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:10.310117 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:10.309886    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:10.310117 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:10.309994    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:10.310117 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:10.310083    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:11.310657 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:11.310618    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:11.311082 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:11.310748    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:12.310846 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.310669    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:12.311230 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.310683    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:12.311230 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:12.310937    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:12.311230 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:12.311008    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:12.399105 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.399073    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-frkc9" event={"ID":"cf1f94f2-24f9-4064-a595-ae2716f9a047","Type":"ContainerStarted","Data":"3f69fa1156a5948e0f56a6615239d9ea92299bed7f7287c000ffafe64517bf30"}
Sep 18 20:21:12.400468 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.400440    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jt29h" event={"ID":"1003b80d-d1eb-4bf7-a93d-96fc66cdd16d","Type":"ContainerStarted","Data":"fb74be878da849870275fbc2d8ece2f69b793943c85aa6c105216927aae1f75d"}
Sep 18 20:21:12.402068 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.402037    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nkg67" event={"ID":"9f0229ef-b6b0-44da-beac-0973f4229671","Type":"ContainerStarted","Data":"83c9edeb9f8c64c977eec1470b7e304a57f4f244be34607b818b0d8177810903"}
Sep 18 20:21:12.403435 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.403411    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms" event={"ID":"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2","Type":"ContainerStarted","Data":"ad4d8d6d6f0df0649bdfefa744e63e226d4fa7a108e6d58e85ed054fc363133c"}
Sep 18 20:21:12.405362 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.405334    2583 generic.go:356] "Generic (PLEG): container finished" podID="4b5e1706-b06b-4fd9-b342-9af08311c3af" containerID="292ab5f156634b33187ac93636fadc0cb223e124bb5171bc2c3eef39107e4707" exitCode=0
Sep 18 20:21:12.405446 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.405409    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p55j2" event={"ID":"4b5e1706-b06b-4fd9-b342-9af08311c3af","Type":"ContainerDied","Data":"292ab5f156634b33187ac93636fadc0cb223e124bb5171bc2c3eef39107e4707"}
Sep 18 20:21:12.407498 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.407479    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dmm78" event={"ID":"4bff1259-e240-4481-a8ea-a6ddc0e0ee6a","Type":"ContainerStarted","Data":"5bbbe657b09b15a934ee2b3e844b37ca5b4eb3e0d4e59c7bd158645f0f5e06f0"}
Sep 18 20:21:12.409079 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.409060    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rlvbw" event={"ID":"86c8b3bd-a424-4004-9c8d-36ae6030370b","Type":"ContainerStarted","Data":"c31d8d5a2a8856d12e47aad7d2d557d117bd89f4af5bd94b0bdac279a3b4ae11"}
Sep 18 20:21:12.411007 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.410991    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:21:12.411393 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.411370    2583 generic.go:356] "Generic (PLEG): container finished" podID="059bd389-2748-46bb-bb0f-6b57cc6d391d" containerID="f435a93e715b7c7d5685d74185f2dacdad57f730870562930dbec1b50623174f" exitCode=1
Sep 18 20:21:12.411442 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.411413    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg" event={"ID":"059bd389-2748-46bb-bb0f-6b57cc6d391d","Type":"ContainerStarted","Data":"5a3bb8a805fbe50538259c41dc4cdca5f43c07278992ef518f35f47ad21ec4aa"}
Sep 18 20:21:12.411476 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.411457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg" event={"ID":"059bd389-2748-46bb-bb0f-6b57cc6d391d","Type":"ContainerDied","Data":"f435a93e715b7c7d5685d74185f2dacdad57f730870562930dbec1b50623174f"}
Sep 18 20:21:12.411505 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.411489    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg" event={"ID":"059bd389-2748-46bb-bb0f-6b57cc6d391d","Type":"ContainerStarted","Data":"3885d9a384c8a6d6ec993b821319cb3c7ab204966f23a0e468e8fe79951b47d7"}
Sep 18 20:21:12.419099 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.419064    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jt29h" podStartSLOduration=3.8082980600000003 podStartE2EDuration="21.419052353s" totalImagesPullingTime="17.610754293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:12.418526806 +0000 UTC m=+23.326429095" watchObservedRunningTime="2026-09-18 20:21:12.419052353 +0000 UTC m=+23.326954631"
Sep 18 20:21:12.432067 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.432028    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dmm78" podStartSLOduration=3.812834371 podStartE2EDuration="21.432014868s" totalImagesPullingTime="17.619180497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:12.431933637 +0000 UTC m=+23.339835920" watchObservedRunningTime="2026-09-18 20:21:12.432014868 +0000 UTC m=+23.339917176"
Sep 18 20:21:12.467193 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.467148    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-frkc9" podStartSLOduration=3.704868169 podStartE2EDuration="21.467116645s" totalImagesPullingTime="17.762248476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:12.467056733 +0000 UTC m=+23.374959012" watchObservedRunningTime="2026-09-18 20:21:12.467116645 +0000 UTC m=+23.375018924"
Sep 18 20:21:12.493792 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.493680    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rlvbw" podStartSLOduration=11.28716157 podStartE2EDuration="21.493660428s" totalImagesPullingTime="10.206498858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:12.493425192 +0000 UTC m=+23.401327500" watchObservedRunningTime="2026-09-18 20:21:12.493660428 +0000 UTC m=+23.401562706"
Sep 18 20:21:12.510954 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:12.510793    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nkg67" podStartSLOduration=3.746419833 podStartE2EDuration="21.510770934s" totalImagesPullingTime="17.764351101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:12.510263111 +0000 UTC m=+23.418165405" watchObservedRunningTime="2026-09-18 20:21:12.510770934 +0000 UTC m=+23.418673277"
Sep 18 20:21:13.250483 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:13.250457    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 20:21:13.259807 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:13.259496    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T20:21:13.250479721Z","UUID":"1e5950ef-037e-43ea-af93-97f6350f7c58","Handler":null,"Name":"","Endpoint":""}
Sep 18 20:21:13.261001 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:13.260984    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 20:21:13.261087 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:13.261014    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 20:21:13.310872 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:13.310840    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:13.311595 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:13.310967    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:13.417109 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:13.417036    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:21:13.417534 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:13.417506    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg" event={"ID":"059bd389-2748-46bb-bb0f-6b57cc6d391d","Type":"ContainerStarted","Data":"f7fc3598ea5091fded856a9cc50b9f8218283c2a06cd4d71552fb350edb6c419"}
Sep 18 20:21:13.417638 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:13.417591    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg" event={"ID":"059bd389-2748-46bb-bb0f-6b57cc6d391d","Type":"ContainerStarted","Data":"d75244ca816dc56071be60bd096955df5f48a6eca59663f0ad492cd887fbd13b"}
Sep 18 20:21:13.417692 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:13.417638    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg" event={"ID":"059bd389-2748-46bb-bb0f-6b57cc6d391d","Type":"ContainerStarted","Data":"9c2e88d3208d68ecdc716764b5f398bf67f50ef820f022fb29e416f58eb4803b"}
Sep 18 20:21:13.419080 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:13.419053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-98jdh" event={"ID":"c011d236-b524-45ce-a873-88ca80a4c3d9","Type":"ContainerStarted","Data":"c53e285122a85c4d050f8f084c1efe2b54d351fe2eae2a76a8e338065d95ad59"}
Sep 18 20:21:13.421303 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:13.421279    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms" event={"ID":"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2","Type":"ContainerStarted","Data":"9e45eaa32ce2ac2e156ff240a777f03ee4014f832eb5dc38e6454372fab7348c"}
Sep 18 20:21:13.434747 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:13.434696    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-98jdh" podStartSLOduration=4.813142956 podStartE2EDuration="22.434684761s" totalImagesPullingTime="17.621541805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:13.434662899 +0000 UTC m=+24.342565178" watchObservedRunningTime="2026-09-18 20:21:13.434684761 +0000 UTC m=+24.342587040"
Sep 18 20:21:14.310016 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:14.309995    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:14.310161 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:14.310082    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:14.310161 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:14.310152    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:14.310250 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:14.310198    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:14.426176 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:14.426139    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms" event={"ID":"e55e670d-adf8-4a8d-8c6c-bf8956b86ae2","Type":"ContainerStarted","Data":"37a6741f6720c5106467e8bde3c0660e517ae2a5dad4bf3c504b13be828f0607"}
Sep 18 20:21:14.435947 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:14.435912    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rlvbw"
Sep 18 20:21:14.436428 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:14.436410    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rlvbw"
Sep 18 20:21:14.443973 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:14.443918    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spsms" podStartSLOduration=3.930587611 podStartE2EDuration="23.443901115s" totalImagesPullingTime="19.513313504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:14.443311958 +0000 UTC m=+25.351214237" watchObservedRunningTime="2026-09-18 20:21:14.443901115 +0000 UTC m=+25.351803395"
Sep 18 20:21:15.310944 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:15.310731    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:15.311105 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:15.311074    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:15.434376 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:15.434348    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:21:15.434826 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:15.434800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg" event={"ID":"059bd389-2748-46bb-bb0f-6b57cc6d391d","Type":"ContainerStarted","Data":"c7a16acba99166ac81ed5726aa5a3f4cf77589d910c207175be131b10a6e56f2"}
Sep 18 20:21:16.310235 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:16.310204    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:16.310437 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:16.310216    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:16.310437 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:16.310323    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:16.310437 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:16.310428    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:17.310414 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:17.310239    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:17.311051 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:17.310505    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:17.441060 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:17.441029    2583 generic.go:356] "Generic (PLEG): container finished" podID="4b5e1706-b06b-4fd9-b342-9af08311c3af" containerID="4f8a2b6f8be00edcbe38090b730f20665dc427a559b7be87e17524d4e0ed2659" exitCode=0
Sep 18 20:21:17.441251 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:17.441132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p55j2" event={"ID":"4b5e1706-b06b-4fd9-b342-9af08311c3af","Type":"ContainerDied","Data":"4f8a2b6f8be00edcbe38090b730f20665dc427a559b7be87e17524d4e0ed2659"}
Sep 18 20:21:17.445051 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:17.445035    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:21:17.445402 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:17.445383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg" event={"ID":"059bd389-2748-46bb-bb0f-6b57cc6d391d","Type":"ContainerStarted","Data":"093b6f5934e5473eae292e218c7de8082077c4367c338bc73630df75c7781502"}
Sep 18 20:21:17.445660 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:17.445643    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:21:17.445802 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:17.445781    2583 scope.go:117] "RemoveContainer" containerID="f435a93e715b7c7d5685d74185f2dacdad57f730870562930dbec1b50623174f"
Sep 18 20:21:17.462367 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:17.462345    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:21:18.057755 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:18.057721    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:21:18.310479 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:18.310401    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:18.310941 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:18.310401    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:18.310941 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:18.310600    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:18.310941 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:18.310502    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:18.451900 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:18.451872    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:21:18.452242 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:18.452221    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg" event={"ID":"059bd389-2748-46bb-bb0f-6b57cc6d391d","Type":"ContainerStarted","Data":"f5dc57310aa210e61a0edd4e66aca96be3108585379cb9c478a4cf9e786d5e5b"}
Sep 18 20:21:18.452401 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:18.452387    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:21:18.468422 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:18.468392    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:21:18.483597 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:18.483549    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg" podStartSLOduration=9.65812994 podStartE2EDuration="27.483535234s" totalImagesPullingTime="17.825405294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:18.482730313 +0000 UTC m=+29.390632591" watchObservedRunningTime="2026-09-18 20:21:18.483535234 +0000 UTC m=+29.391437539"
Sep 18 20:21:19.310112 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:19.310075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:19.310341 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:19.310235    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:19.458689 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:19.458649    2583 generic.go:356] "Generic (PLEG): container finished" podID="4b5e1706-b06b-4fd9-b342-9af08311c3af" containerID="447c2d2a55872fb1285adde749ce59ea97459bf19905e665510ab33fda8e06ea" exitCode=0
Sep 18 20:21:19.459184 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:19.458738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p55j2" event={"ID":"4b5e1706-b06b-4fd9-b342-9af08311c3af","Type":"ContainerDied","Data":"447c2d2a55872fb1285adde749ce59ea97459bf19905e665510ab33fda8e06ea"}
Sep 18 20:21:20.310230 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:20.310193    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:20.310390 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:20.310289    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:20.310390 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:20.310345    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:20.310469 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:20.310426    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:21.076156 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:21.076113    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rlvbw"
Sep 18 20:21:21.076616 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:21.076236    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 20:21:21.076810 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:21.076792    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rlvbw"
Sep 18 20:21:21.311168 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:21.311136    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:21.311345 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:21.311222    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:21.465161 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:21.465105    2583 generic.go:356] "Generic (PLEG): container finished" podID="4b5e1706-b06b-4fd9-b342-9af08311c3af" containerID="e6a19bd30586a87d4c2a9909282a587c5358e9561f110b3353e14e288a1ef63c" exitCode=0
Sep 18 20:21:21.465339 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:21.465191    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p55j2" event={"ID":"4b5e1706-b06b-4fd9-b342-9af08311c3af","Type":"ContainerDied","Data":"e6a19bd30586a87d4c2a9909282a587c5358e9561f110b3353e14e288a1ef63c"}
Sep 18 20:21:22.310021 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:22.309989    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:22.310485 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:22.309989    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:22.310485 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:22.310085    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:22.310485 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:22.310200    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:23.310187 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:23.310155    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:23.310584 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:23.310311    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:24.309888 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:24.309848    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:24.310090 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:24.309856    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:24.310090 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:24.309981    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:24.310090 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:24.310041    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:24.949731 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:24.949695    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:24.950017 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:24.949761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:24.950017 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:24.949862    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:21:24.950017 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:24.949892    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:21:24.950017 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:24.949936    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs podName:eaf22891-e55a-433a-a689-fb5b3e3575f9 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:56.949916367 +0000 UTC m=+67.857818646 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs") pod "network-metrics-daemon-wkf7p" (UID: "eaf22891-e55a-433a-a689-fb5b3e3575f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:21:24.950017 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:24.949955    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret podName:941899a6-9517-410e-b439-94c967f6c162 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:56.949948205 +0000 UTC m=+67.857850462 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret") pod "global-pull-secret-syncer-9rj8d" (UID: "941899a6-9517-410e-b439-94c967f6c162") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:21:25.050360 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:25.050323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pp4l\" (UniqueName: \"kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l\") pod \"network-check-target-qvkkm\" (UID: \"1ecf386e-6ab3-4448-9c76-75a96b3cfa75\") " pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:25.050525 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:25.050472    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:21:25.050525 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:25.050493    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:21:25.050525 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:25.050508    2583 projected.go:196] Error preparing data for projected volume kube-api-access-6pp4l for pod openshift-network-diagnostics/network-check-target-qvkkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:21:25.050652 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:25.050568    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l podName:1ecf386e-6ab3-4448-9c76-75a96b3cfa75 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:57.05054924 +0000 UTC m=+67.958451501 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6pp4l" (UniqueName: "kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l") pod "network-check-target-qvkkm" (UID: "1ecf386e-6ab3-4448-9c76-75a96b3cfa75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:21:25.310554 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:25.310476    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:25.310963 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:25.310627    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:26.310367 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:26.310336    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:26.310543 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:26.310336    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:26.310543 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:26.310453    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:26.310871 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:26.310552    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:26.479797 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:26.479766    2583 generic.go:356] "Generic (PLEG): container finished" podID="4b5e1706-b06b-4fd9-b342-9af08311c3af" containerID="5c2f308b52d407a3369d9d2485fb15152fc3b17f1287aae3575beeebcf50d1a3" exitCode=0
Sep 18 20:21:26.479985 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:26.479807    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p55j2" event={"ID":"4b5e1706-b06b-4fd9-b342-9af08311c3af","Type":"ContainerDied","Data":"5c2f308b52d407a3369d9d2485fb15152fc3b17f1287aae3575beeebcf50d1a3"}
Sep 18 20:21:27.309940 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:27.309768    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:27.310155 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:27.310015    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:27.485691 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:27.485658    2583 generic.go:356] "Generic (PLEG): container finished" podID="4b5e1706-b06b-4fd9-b342-9af08311c3af" containerID="b54d1a3c46d4c05ab39131c90aa644407ce706ebb4d237324ef1a7e909f7ca23" exitCode=0
Sep 18 20:21:27.486153 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:27.485707    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p55j2" event={"ID":"4b5e1706-b06b-4fd9-b342-9af08311c3af","Type":"ContainerDied","Data":"b54d1a3c46d4c05ab39131c90aa644407ce706ebb4d237324ef1a7e909f7ca23"}
Sep 18 20:21:28.309946 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:28.309912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:28.310137 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:28.310006    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:28.310137 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:28.310032    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:28.310137 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:28.310080    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:28.491540 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:28.491510    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p55j2" event={"ID":"4b5e1706-b06b-4fd9-b342-9af08311c3af","Type":"ContainerStarted","Data":"f40ead111c4722a8b30ab421a06ccc827c760642de9243651f93fc38d298db37"}
Sep 18 20:21:28.860464 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:28.860391    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p55j2" podStartSLOduration=9.318074621 podStartE2EDuration="37.860364185s" totalImagesPullingTime="28.542289564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:28.517468296 +0000 UTC m=+39.425370575" watchObservedRunningTime="2026-09-18 20:21:28.860364185 +0000 UTC m=+39.768266467"
Sep 18 20:21:28.860691 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:28.860673    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wkf7p"]
Sep 18 20:21:28.860869 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:28.860853    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:28.861010 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:28.860987    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:28.865512 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:28.865472    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9rj8d"]
Sep 18 20:21:28.865619 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:28.865536    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvkkm"]
Sep 18 20:21:28.865619 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:28.865613    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:28.865736 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:28.865705    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:28.865932 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:28.865902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:28.866049 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:28.865986    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:30.309725 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:30.309693    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:30.310369 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:30.309793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:30.310369 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:30.309804    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:30.310369 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:30.309883    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:30.310369 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:30.309920    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:30.310369 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:30.309974    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:32.310353 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:32.310312    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:32.310783 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:32.310331    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:32.310783 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:32.310428    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:32.310783 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:32.310430    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rj8d" podUID="941899a6-9517-410e-b439-94c967f6c162"
Sep 18 20:21:32.310783 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:32.310492    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvkkm" podUID="1ecf386e-6ab3-4448-9c76-75a96b3cfa75"
Sep 18 20:21:32.310783 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:32.310578    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkf7p" podUID="eaf22891-e55a-433a-a689-fb5b3e3575f9"
Sep 18 20:21:33.981648 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:33.981618    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeReady"
Sep 18 20:21:33.982011 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:33.981734    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 20:21:34.042685 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.042663    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k6v7r"]
Sep 18 20:21:34.044766 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.044746    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.045391 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.045370    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-plv2v"]
Sep 18 20:21:34.047433 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.047413    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 20:21:34.047534 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.047455    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-plv2v"
Sep 18 20:21:34.047593 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.047411    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l94f7\""
Sep 18 20:21:34.047645 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.047410    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 20:21:34.050268 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.050248    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 20:21:34.050874 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.050744    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cgdxn\""
Sep 18 20:21:34.050874 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.050760    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 20:21:34.054474 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.053429    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 20:21:34.058289 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.058273    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-plv2v"]
Sep 18 20:21:34.063604 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.063578    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k6v7r"]
Sep 18 20:21:34.074111 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.074091    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dfc68"]
Sep 18 20:21:34.075893 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.075877    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.079649 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.079631    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s6fww\""
Sep 18 20:21:34.079649 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.079647    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 20:21:34.079781 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.079667    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 20:21:34.079781 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.079694    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 20:21:34.079955 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.079943    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 20:21:34.091342 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.091317    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dfc68"]
Sep 18 20:21:34.213472 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.213437    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f7cd4bd84-pjnmf"]
Sep 18 20:21:34.213615 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.213592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a824cb57-ceee-40c5-927e-d3c6bbc13df4-crio-socket\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.213662 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.213623    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e375926-9dc6-49d2-865b-fa57f024b925-metrics-tls\") pod \"dns-default-k6v7r\" (UID: \"3e375926-9dc6-49d2-865b-fa57f024b925\") " pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.213662 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.213641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e375926-9dc6-49d2-865b-fa57f024b925-config-volume\") pod \"dns-default-k6v7r\" (UID: \"3e375926-9dc6-49d2-865b-fa57f024b925\") " pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.213763 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.213660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a824cb57-ceee-40c5-927e-d3c6bbc13df4-data-volume\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.213763 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.213714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a824cb57-ceee-40c5-927e-d3c6bbc13df4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.213859 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.213814    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e375926-9dc6-49d2-865b-fa57f024b925-tmp-dir\") pod \"dns-default-k6v7r\" (UID: \"3e375926-9dc6-49d2-865b-fa57f024b925\") " pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.213859 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.213848    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a824cb57-ceee-40c5-927e-d3c6bbc13df4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.213957 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.213874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq42r\" (UniqueName: \"kubernetes.io/projected/a824cb57-ceee-40c5-927e-d3c6bbc13df4-kube-api-access-pq42r\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.213957 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.213906    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj7f5\" (UniqueName: \"kubernetes.io/projected/9ddba8b7-bff2-4785-9db8-ddd7c810c350-kube-api-access-dj7f5\") pod \"ingress-canary-plv2v\" (UID: \"9ddba8b7-bff2-4785-9db8-ddd7c810c350\") " pod="openshift-ingress-canary/ingress-canary-plv2v"
Sep 18 20:21:34.213957 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.213935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pzqw\" (UniqueName: \"kubernetes.io/projected/3e375926-9dc6-49d2-865b-fa57f024b925-kube-api-access-2pzqw\") pod \"dns-default-k6v7r\" (UID: \"3e375926-9dc6-49d2-865b-fa57f024b925\") " pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.214074 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.213971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ddba8b7-bff2-4785-9db8-ddd7c810c350-cert\") pod \"ingress-canary-plv2v\" (UID: \"9ddba8b7-bff2-4785-9db8-ddd7c810c350\") " pod="openshift-ingress-canary/ingress-canary-plv2v"
Sep 18 20:21:34.215705 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.215691    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.218384 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.218365    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 20:21:34.218496 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.218479    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 20:21:34.218560 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.218520    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 20:21:34.218636 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.218622    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 20:21:34.218636 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.218629    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 20:21:34.218733 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.218686    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 20:21:34.218949 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.218926    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 20:21:34.220006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.219991    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bqckb\""
Sep 18 20:21:34.224148 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.224113    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 20:21:34.228903 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.228882    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f7cd4bd84-pjnmf"]
Sep 18 20:21:34.310715 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.310659    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:34.310830 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.310744    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:34.310830 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.310753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:34.313761 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.313733    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 20:21:34.313871 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.313771    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 20:21:34.313871 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.313835    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 20:21:34.313871 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.313835    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 20:21:34.313871 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.313862    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ls8xl\""
Sep 18 20:21:34.314026 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.313974    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5lg4r\""
Sep 18 20:21:34.314223 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ddba8b7-bff2-4785-9db8-ddd7c810c350-cert\") pod \"ingress-canary-plv2v\" (UID: \"9ddba8b7-bff2-4785-9db8-ddd7c810c350\") " pod="openshift-ingress-canary/ingress-canary-plv2v"
Sep 18 20:21:34.314283 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-oauth-serving-cert\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.314283 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a824cb57-ceee-40c5-927e-d3c6bbc13df4-crio-socket\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.314283 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-trusted-ca-bundle\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.314283 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e375926-9dc6-49d2-865b-fa57f024b925-metrics-tls\") pod \"dns-default-k6v7r\" (UID: \"3e375926-9dc6-49d2-865b-fa57f024b925\") " pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.314451 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e375926-9dc6-49d2-865b-fa57f024b925-config-volume\") pod \"dns-default-k6v7r\" (UID: \"3e375926-9dc6-49d2-865b-fa57f024b925\") " pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.314503 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-console-config\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.314503 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314474    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-service-ca\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.314503 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a824cb57-ceee-40c5-927e-d3c6bbc13df4-data-volume\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.314642 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a824cb57-ceee-40c5-927e-d3c6bbc13df4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.314642 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a824cb57-ceee-40c5-927e-d3c6bbc13df4-crio-socket\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.314642 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314562    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfe139a5-1a94-4614-aedd-34016aa65250-console-oauth-config\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.314642 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e375926-9dc6-49d2-865b-fa57f024b925-tmp-dir\") pod \"dns-default-k6v7r\" (UID: \"3e375926-9dc6-49d2-865b-fa57f024b925\") " pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.314642 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a824cb57-ceee-40c5-927e-d3c6bbc13df4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.314642 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq42r\" (UniqueName: \"kubernetes.io/projected/a824cb57-ceee-40c5-927e-d3c6bbc13df4-kube-api-access-pq42r\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.314913 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj7f5\" (UniqueName: \"kubernetes.io/projected/9ddba8b7-bff2-4785-9db8-ddd7c810c350-kube-api-access-dj7f5\") pod \"ingress-canary-plv2v\" (UID: \"9ddba8b7-bff2-4785-9db8-ddd7c810c350\") " pod="openshift-ingress-canary/ingress-canary-plv2v"
Sep 18 20:21:34.314913 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfe139a5-1a94-4614-aedd-34016aa65250-console-serving-cert\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.314913 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46t5n\" (UniqueName: \"kubernetes.io/projected/cfe139a5-1a94-4614-aedd-34016aa65250-kube-api-access-46t5n\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.314913 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pzqw\" (UniqueName: \"kubernetes.io/projected/3e375926-9dc6-49d2-865b-fa57f024b925-kube-api-access-2pzqw\") pod \"dns-default-k6v7r\" (UID: \"3e375926-9dc6-49d2-865b-fa57f024b925\") " pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.314913 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.314843    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a824cb57-ceee-40c5-927e-d3c6bbc13df4-data-volume\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.315248 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.315028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e375926-9dc6-49d2-865b-fa57f024b925-config-volume\") pod \"dns-default-k6v7r\" (UID: \"3e375926-9dc6-49d2-865b-fa57f024b925\") " pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.315526 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.315359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e375926-9dc6-49d2-865b-fa57f024b925-tmp-dir\") pod \"dns-default-k6v7r\" (UID: \"3e375926-9dc6-49d2-865b-fa57f024b925\") " pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.315621 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.315387    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a824cb57-ceee-40c5-927e-d3c6bbc13df4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.319095 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.319077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e375926-9dc6-49d2-865b-fa57f024b925-metrics-tls\") pod \"dns-default-k6v7r\" (UID: \"3e375926-9dc6-49d2-865b-fa57f024b925\") " pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.319279 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.319098    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a824cb57-ceee-40c5-927e-d3c6bbc13df4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.319279 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.319248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ddba8b7-bff2-4785-9db8-ddd7c810c350-cert\") pod \"ingress-canary-plv2v\" (UID: \"9ddba8b7-bff2-4785-9db8-ddd7c810c350\") " pod="openshift-ingress-canary/ingress-canary-plv2v"
Sep 18 20:21:34.322787 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.322765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq42r\" (UniqueName: \"kubernetes.io/projected/a824cb57-ceee-40c5-927e-d3c6bbc13df4-kube-api-access-pq42r\") pod \"insights-runtime-extractor-dfc68\" (UID: \"a824cb57-ceee-40c5-927e-d3c6bbc13df4\") " pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.322859 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.322776    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj7f5\" (UniqueName: \"kubernetes.io/projected/9ddba8b7-bff2-4785-9db8-ddd7c810c350-kube-api-access-dj7f5\") pod \"ingress-canary-plv2v\" (UID: \"9ddba8b7-bff2-4785-9db8-ddd7c810c350\") " pod="openshift-ingress-canary/ingress-canary-plv2v"
Sep 18 20:21:34.322859 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.322837    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pzqw\" (UniqueName: \"kubernetes.io/projected/3e375926-9dc6-49d2-865b-fa57f024b925-kube-api-access-2pzqw\") pod \"dns-default-k6v7r\" (UID: \"3e375926-9dc6-49d2-865b-fa57f024b925\") " pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.358812 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.358787    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:34.362815 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.362800    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-plv2v"
Sep 18 20:21:34.384748 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.384720    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dfc68"
Sep 18 20:21:34.415145 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.415100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfe139a5-1a94-4614-aedd-34016aa65250-console-oauth-config\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.415288 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.415211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfe139a5-1a94-4614-aedd-34016aa65250-console-serving-cert\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.415353 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.415240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46t5n\" (UniqueName: \"kubernetes.io/projected/cfe139a5-1a94-4614-aedd-34016aa65250-kube-api-access-46t5n\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.415656 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.415632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-oauth-serving-cert\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.415786 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.415673    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-trusted-ca-bundle\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.415786 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.415713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-console-config\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.415786 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.415736    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-service-ca\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.417420 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.417392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-trusted-ca-bundle\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.418583 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.417895    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-oauth-serving-cert\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.418583 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.418381    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-console-config\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.418735 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.418637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-service-ca\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.421637 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.419981    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfe139a5-1a94-4614-aedd-34016aa65250-console-oauth-config\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.422628 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.422228    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfe139a5-1a94-4614-aedd-34016aa65250-console-serving-cert\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.424924 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.424902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46t5n\" (UniqueName: \"kubernetes.io/projected/cfe139a5-1a94-4614-aedd-34016aa65250-kube-api-access-46t5n\") pod \"console-7f7cd4bd84-pjnmf\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") " pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.524228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.524197    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:34.651185 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.651153    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k6v7r"]
Sep 18 20:21:34.655323 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:21:34.655296    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e375926_9dc6_49d2_865b_fa57f024b925.slice/crio-112e6b810d2f006127cc54a6938e06a3153b2338936dd0507c25eb77792527d2 WatchSource:0}: Error finding container 112e6b810d2f006127cc54a6938e06a3153b2338936dd0507c25eb77792527d2: Status 404 returned error can't find the container with id 112e6b810d2f006127cc54a6938e06a3153b2338936dd0507c25eb77792527d2
Sep 18 20:21:34.655565 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.655544    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-plv2v"]
Sep 18 20:21:34.657398 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:21:34.657375    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ddba8b7_bff2_4785_9db8_ddd7c810c350.slice/crio-6231728f8e79352daf6e7eea28abb55639b36fbca401002429e567b7ced79285 WatchSource:0}: Error finding container 6231728f8e79352daf6e7eea28abb55639b36fbca401002429e567b7ced79285: Status 404 returned error can't find the container with id 6231728f8e79352daf6e7eea28abb55639b36fbca401002429e567b7ced79285
Sep 18 20:21:34.670742 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.670718    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dfc68"]
Sep 18 20:21:34.674249 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:21:34.674227    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda824cb57_ceee_40c5_927e_d3c6bbc13df4.slice/crio-8bb4db30ad1d011717c14504cd36d349a162efc2feefe5078d47c689e082ed1c WatchSource:0}: Error finding container 8bb4db30ad1d011717c14504cd36d349a162efc2feefe5078d47c689e082ed1c: Status 404 returned error can't find the container with id 8bb4db30ad1d011717c14504cd36d349a162efc2feefe5078d47c689e082ed1c
Sep 18 20:21:34.791691 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:34.791660    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f7cd4bd84-pjnmf"]
Sep 18 20:21:34.795312 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:21:34.795289    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe139a5_1a94_4614_aedd_34016aa65250.slice/crio-2df22ca6f9726566090e70dc3a8dfbe73a8a9b09e1b216feb14cd35bd94e277f WatchSource:0}: Error finding container 2df22ca6f9726566090e70dc3a8dfbe73a8a9b09e1b216feb14cd35bd94e277f: Status 404 returned error can't find the container with id 2df22ca6f9726566090e70dc3a8dfbe73a8a9b09e1b216feb14cd35bd94e277f
Sep 18 20:21:35.509823 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.509742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7cd4bd84-pjnmf" event={"ID":"cfe139a5-1a94-4614-aedd-34016aa65250","Type":"ContainerStarted","Data":"2df22ca6f9726566090e70dc3a8dfbe73a8a9b09e1b216feb14cd35bd94e277f"}
Sep 18 20:21:35.515511 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.515423    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-plv2v" event={"ID":"9ddba8b7-bff2-4785-9db8-ddd7c810c350","Type":"ContainerStarted","Data":"6231728f8e79352daf6e7eea28abb55639b36fbca401002429e567b7ced79285"}
Sep 18 20:21:35.518249 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.518171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k6v7r" event={"ID":"3e375926-9dc6-49d2-865b-fa57f024b925","Type":"ContainerStarted","Data":"112e6b810d2f006127cc54a6938e06a3153b2338936dd0507c25eb77792527d2"}
Sep 18 20:21:35.520883 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.520771    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dfc68" event={"ID":"a824cb57-ceee-40c5-927e-d3c6bbc13df4","Type":"ContainerStarted","Data":"7649ecc2dc4f80c06e1acb99d9b5d476a66755ceafbc43c2685a01470a595763"}
Sep 18 20:21:35.520883 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.520822    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dfc68" event={"ID":"a824cb57-ceee-40c5-927e-d3c6bbc13df4","Type":"ContainerStarted","Data":"8bb4db30ad1d011717c14504cd36d349a162efc2feefe5078d47c689e082ed1c"}
Sep 18 20:21:35.649157 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.648277    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"]
Sep 18 20:21:35.653740 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.651920    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.655147 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.655094    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cwhmr\""
Sep 18 20:21:35.655494 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.655476    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 20:21:35.655761 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.655712    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 20:21:35.657449 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.657428    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 20:21:35.657684 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.657667    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 20:21:35.657998 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.657975    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 20:21:35.667311 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.666841    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gp7m9"]
Sep 18 20:21:35.678348 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.673772    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"]
Sep 18 20:21:35.678348 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.673822    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"]
Sep 18 20:21:35.678348 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.673898    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.678348 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.677862    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 20:21:35.678747 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.678680    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 20:21:35.678747 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.678687    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ssg9g\""
Sep 18 20:21:35.678883 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.678853    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 20:21:35.679634 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.679546    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.685565 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.684273    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 20:21:35.685565 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.684542    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 20:21:35.685565 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.684605    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"]
Sep 18 20:21:35.685565 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.684742    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x9hqk\""
Sep 18 20:21:35.685565 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.684915    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 20:21:35.829682 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00bf8807-5f69-4594-b985-f46b1423b94c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-pw4h6\" (UID: \"00bf8807-5f69-4594-b985-f46b1423b94c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.829682 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829655    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9531a7cb-4020-4d03-bde2-df7edfb474ec-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.829849 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9531a7cb-4020-4d03-bde2-df7edfb474ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.829849 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-tls\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.829849 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00bf8807-5f69-4594-b985-f46b1423b94c-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-pw4h6\" (UID: \"00bf8807-5f69-4594-b985-f46b1423b94c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.829849 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829773    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6967m\" (UniqueName: \"kubernetes.io/projected/d7ae1a48-dec2-4ae8-9e44-02402f265518-kube-api-access-6967m\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.829849 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-wtmp\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.829849 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829839    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvzf\" (UniqueName: \"kubernetes.io/projected/9531a7cb-4020-4d03-bde2-df7edfb474ec-kube-api-access-swvzf\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.830020 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7ae1a48-dec2-4ae8-9e44-02402f265518-sys\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.830020 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829889    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/00bf8807-5f69-4594-b985-f46b1423b94c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-pw4h6\" (UID: \"00bf8807-5f69-4594-b985-f46b1423b94c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.830020 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829917    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9531a7cb-4020-4d03-bde2-df7edfb474ec-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.830020 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829946    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.830020 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.829974    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj7v5\" (UniqueName: \"kubernetes.io/projected/00bf8807-5f69-4594-b985-f46b1423b94c-kube-api-access-jj7v5\") pod \"openshift-state-metrics-86c5bc79d6-pw4h6\" (UID: \"00bf8807-5f69-4594-b985-f46b1423b94c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.830020 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.830002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9531a7cb-4020-4d03-bde2-df7edfb474ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.830253 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.830028    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-accelerators-collector-config\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.830253 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.830087    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9531a7cb-4020-4d03-bde2-df7edfb474ec-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.830253 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.830111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7ae1a48-dec2-4ae8-9e44-02402f265518-metrics-client-ca\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.830253 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.830160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-textfile\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.830417 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.830283    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7ae1a48-dec2-4ae8-9e44-02402f265518-root\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.931643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.931696    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj7v5\" (UniqueName: \"kubernetes.io/projected/00bf8807-5f69-4594-b985-f46b1423b94c-kube-api-access-jj7v5\") pod \"openshift-state-metrics-86c5bc79d6-pw4h6\" (UID: \"00bf8807-5f69-4594-b985-f46b1423b94c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.931717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9531a7cb-4020-4d03-bde2-df7edfb474ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.931742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-accelerators-collector-config\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.931801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9531a7cb-4020-4d03-bde2-df7edfb474ec-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.931819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7ae1a48-dec2-4ae8-9e44-02402f265518-metrics-client-ca\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.931838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-textfile\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.931882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7ae1a48-dec2-4ae8-9e44-02402f265518-root\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.931916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00bf8807-5f69-4594-b985-f46b1423b94c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-pw4h6\" (UID: \"00bf8807-5f69-4594-b985-f46b1423b94c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.931943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9531a7cb-4020-4d03-bde2-df7edfb474ec-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.931969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9531a7cb-4020-4d03-bde2-df7edfb474ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.932017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-tls\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.932048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00bf8807-5f69-4594-b985-f46b1423b94c-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-pw4h6\" (UID: \"00bf8807-5f69-4594-b985-f46b1423b94c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.932077    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6967m\" (UniqueName: \"kubernetes.io/projected/d7ae1a48-dec2-4ae8-9e44-02402f265518-kube-api-access-6967m\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.932110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-wtmp\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.933297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.932169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvzf\" (UniqueName: \"kubernetes.io/projected/9531a7cb-4020-4d03-bde2-df7edfb474ec-kube-api-access-swvzf\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.934169 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.932192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7ae1a48-dec2-4ae8-9e44-02402f265518-sys\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.934169 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.932215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/00bf8807-5f69-4594-b985-f46b1423b94c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-pw4h6\" (UID: \"00bf8807-5f69-4594-b985-f46b1423b94c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.934169 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.932241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9531a7cb-4020-4d03-bde2-df7edfb474ec-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.936628 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.934790    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-accelerators-collector-config\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.936628 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.935177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-wtmp\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.936628 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:35.935375    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 20:21:35.936628 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:35.935454    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-tls podName:d7ae1a48-dec2-4ae8-9e44-02402f265518 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:36.435434568 +0000 UTC m=+47.343336845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-tls") pod "node-exporter-gp7m9" (UID: "d7ae1a48-dec2-4ae8-9e44-02402f265518") : secret "node-exporter-tls" not found
Sep 18 20:21:35.936628 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.935762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7ae1a48-dec2-4ae8-9e44-02402f265518-metrics-client-ca\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.936628 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.935819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9531a7cb-4020-4d03-bde2-df7edfb474ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.936628 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.935888    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7ae1a48-dec2-4ae8-9e44-02402f265518-sys\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.936628 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.935977    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9531a7cb-4020-4d03-bde2-df7edfb474ec-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.936628 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.936090    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9531a7cb-4020-4d03-bde2-df7edfb474ec-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.936628 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.936184    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7ae1a48-dec2-4ae8-9e44-02402f265518-root\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.936628 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.936229    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-textfile\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.937943 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.937920    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00bf8807-5f69-4594-b985-f46b1423b94c-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-pw4h6\" (UID: \"00bf8807-5f69-4594-b985-f46b1423b94c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.938527 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.938440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9531a7cb-4020-4d03-bde2-df7edfb474ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.941356 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.939385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/00bf8807-5f69-4594-b985-f46b1423b94c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-pw4h6\" (UID: \"00bf8807-5f69-4594-b985-f46b1423b94c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.941356 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.939948    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9531a7cb-4020-4d03-bde2-df7edfb474ec-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.941868 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.941779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.942821 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.942741    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00bf8807-5f69-4594-b985-f46b1423b94c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-pw4h6\" (UID: \"00bf8807-5f69-4594-b985-f46b1423b94c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.947291 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.947250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj7v5\" (UniqueName: \"kubernetes.io/projected/00bf8807-5f69-4594-b985-f46b1423b94c-kube-api-access-jj7v5\") pod \"openshift-state-metrics-86c5bc79d6-pw4h6\" (UID: \"00bf8807-5f69-4594-b985-f46b1423b94c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.950906 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.950826    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvzf\" (UniqueName: \"kubernetes.io/projected/9531a7cb-4020-4d03-bde2-df7edfb474ec-kube-api-access-swvzf\") pod \"kube-state-metrics-84c885658c-6cdhw\" (UID: \"9531a7cb-4020-4d03-bde2-df7edfb474ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:35.951583 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.951543    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6967m\" (UniqueName: \"kubernetes.io/projected/d7ae1a48-dec2-4ae8-9e44-02402f265518-kube-api-access-6967m\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:35.967138 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.966881    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"
Sep 18 20:21:35.999949 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:35.999714    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"
Sep 18 20:21:36.257712 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.257568    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6"]
Sep 18 20:21:36.264609 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:21:36.264555    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bf8807_5f69_4594_b985_f46b1423b94c.slice/crio-643a43b2a17a7d6fa21293ad170feab8f6e374725f01b2315d3f2482efbccc36 WatchSource:0}: Error finding container 643a43b2a17a7d6fa21293ad170feab8f6e374725f01b2315d3f2482efbccc36: Status 404 returned error can't find the container with id 643a43b2a17a7d6fa21293ad170feab8f6e374725f01b2315d3f2482efbccc36
Sep 18 20:21:36.321839 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.321627    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-6cdhw"]
Sep 18 20:21:36.437824 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.437787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-tls\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:36.441809 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.441780    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7ae1a48-dec2-4ae8-9e44-02402f265518-node-exporter-tls\") pod \"node-exporter-gp7m9\" (UID: \"d7ae1a48-dec2-4ae8-9e44-02402f265518\") " pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:36.533336 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.533253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dfc68" event={"ID":"a824cb57-ceee-40c5-927e-d3c6bbc13df4","Type":"ContainerStarted","Data":"330517c1ba1787ff4f703481c70e700f94b8359aa427c1eb3ffe5e9c7f7d80c1"}
Sep 18 20:21:36.538405 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.538369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6" event={"ID":"00bf8807-5f69-4594-b985-f46b1423b94c","Type":"ContainerStarted","Data":"ed1e0989b83e240c21f315d33c5b0d4bb209e8235762afcd2bd8ca7bc3b008d0"}
Sep 18 20:21:36.538535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.538430    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6" event={"ID":"00bf8807-5f69-4594-b985-f46b1423b94c","Type":"ContainerStarted","Data":"643a43b2a17a7d6fa21293ad170feab8f6e374725f01b2315d3f2482efbccc36"}
Sep 18 20:21:36.593232 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.593172    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gp7m9"
Sep 18 20:21:36.732200 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.730870    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:21:36.751272 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.750276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.751272 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.750449    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:21:36.754006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.753162    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 20:21:36.754006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.753435    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 20:21:36.754006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.753564    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 20:21:36.754006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.753750    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 20:21:36.754006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.753790    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 20:21:36.754006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.753987    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 20:21:36.755528 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.754841    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 20:21:36.755528 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.754843    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 20:21:36.755528 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.755251    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 20:21:36.756085 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.755847    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gtw5k\""
Sep 18 20:21:36.841462 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841307    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.841462 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841347    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.841462 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ceb26837-53a9-42bc-b65c-bd96eea63443-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.841462 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841386    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.841462 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-config-volume\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.841830 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceb26837-53a9-42bc-b65c-bd96eea63443-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.841830 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841544    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ceb26837-53a9-42bc-b65c-bd96eea63443-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.841830 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.841830 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ceb26837-53a9-42bc-b65c-bd96eea63443-config-out\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.841830 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.841830 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841766    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-web-config\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.841830 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841796    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb26837-53a9-42bc-b65c-bd96eea63443-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.841830 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.841816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h84wf\" (UniqueName: \"kubernetes.io/projected/ceb26837-53a9-42bc-b65c-bd96eea63443-kube-api-access-h84wf\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.942874 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.942784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.942874 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.942839    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ceb26837-53a9-42bc-b65c-bd96eea63443-config-out\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.942874 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.942870    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.943166 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.942922    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-web-config\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.943166 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.942952    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb26837-53a9-42bc-b65c-bd96eea63443-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.943166 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.942981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h84wf\" (UniqueName: \"kubernetes.io/projected/ceb26837-53a9-42bc-b65c-bd96eea63443-kube-api-access-h84wf\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.943166 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.943016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.943166 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.943063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.943166 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.943096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ceb26837-53a9-42bc-b65c-bd96eea63443-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.943166 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.943147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.943508 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.943172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-config-volume\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.943508 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.943195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceb26837-53a9-42bc-b65c-bd96eea63443-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.943508 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.943247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ceb26837-53a9-42bc-b65c-bd96eea63443-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.944117 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:36.944091    2583 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 18 20:21:36.944273 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:36.944186    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-main-tls podName:ceb26837-53a9-42bc-b65c-bd96eea63443 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:37.444166612 +0000 UTC m=+48.352068870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443") : secret "alertmanager-main-tls" not found
Sep 18 20:21:36.944927 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.944898    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ceb26837-53a9-42bc-b65c-bd96eea63443-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.946383 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.946351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb26837-53a9-42bc-b65c-bd96eea63443-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.947211 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.947186    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceb26837-53a9-42bc-b65c-bd96eea63443-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.950756 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.950551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.950756 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.950633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-web-config\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.950756 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.950633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-config-volume\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.951294 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.951271    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ceb26837-53a9-42bc-b65c-bd96eea63443-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.951400 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.951387    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.952418 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.952392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.954662 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.953048    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.954852 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.954808    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h84wf\" (UniqueName: \"kubernetes.io/projected/ceb26837-53a9-42bc-b65c-bd96eea63443-kube-api-access-h84wf\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:36.956815 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:36.956772    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ceb26837-53a9-42bc-b65c-bd96eea63443-config-out\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:37.189673 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:21:37.189639    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9531a7cb_4020_4d03_bde2_df7edfb474ec.slice/crio-58f47f87c009f743bf8075e3994ecc092b7b3720cf99b56a7625bf832e074a1d WatchSource:0}: Error finding container 58f47f87c009f743bf8075e3994ecc092b7b3720cf99b56a7625bf832e074a1d: Status 404 returned error can't find the container with id 58f47f87c009f743bf8075e3994ecc092b7b3720cf99b56a7625bf832e074a1d
Sep 18 20:21:37.447781 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.447703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:37.450503 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.450479    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:37.543044 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.543000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw" event={"ID":"9531a7cb-4020-4d03-bde2-df7edfb474ec","Type":"ContainerStarted","Data":"58f47f87c009f743bf8075e3994ecc092b7b3720cf99b56a7625bf832e074a1d"}
Sep 18 20:21:37.630257 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.630223    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-85d85db47d-29rrf"]
Sep 18 20:21:37.643003 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.642981    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.644145 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.644105    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85d85db47d-29rrf"]
Sep 18 20:21:37.645991 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.645904    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 20:21:37.646101 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.646027    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-79kjplvpnj4nf\""
Sep 18 20:21:37.646101 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.646082    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 20:21:37.646454 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.646157    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 20:21:37.646454 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.646252    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 20:21:37.646454 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.646341    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 20:21:37.646454 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.646357    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x5xnc\""
Sep 18 20:21:37.648965 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.648946    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.649063 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.648975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ed5f4a-9853-4d69-872f-cfb6074afecc-metrics-client-ca\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.649063 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.649003    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.649191 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.649072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-tls\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.649191 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.649095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgstv\" (UniqueName: \"kubernetes.io/projected/f1ed5f4a-9853-4d69-872f-cfb6074afecc-kube-api-access-fgstv\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.649191 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.649147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.649191 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.649173    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-grpc-tls\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.649353 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.649221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.664559 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.664541    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:21:37.749929 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.749863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.749929 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.749915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.750133 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.749943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ed5f4a-9853-4d69-872f-cfb6074afecc-metrics-client-ca\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.750133 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.749965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.750133 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.750027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-tls\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.750133 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.750050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgstv\" (UniqueName: \"kubernetes.io/projected/f1ed5f4a-9853-4d69-872f-cfb6074afecc-kube-api-access-fgstv\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.750133 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.750080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.750133 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.750104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-grpc-tls\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.750805 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.750746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ed5f4a-9853-4d69-872f-cfb6074afecc-metrics-client-ca\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.752750 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.752724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.753100 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.752960    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.753199 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.753108    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.753258 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.753232    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.753307 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.753261    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-thanos-querier-tls\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.753406 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.753390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1ed5f4a-9853-4d69-872f-cfb6074afecc-secret-grpc-tls\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.761088 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.761069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgstv\" (UniqueName: \"kubernetes.io/projected/f1ed5f4a-9853-4d69-872f-cfb6074afecc-kube-api-access-fgstv\") pod \"thanos-querier-85d85db47d-29rrf\" (UID: \"f1ed5f4a-9853-4d69-872f-cfb6074afecc\") " pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:37.953037 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:37.953002    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:39.955574 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:39.955541    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-847f7f5f44-wqqzc"]
Sep 18 20:21:39.982520 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:39.982490    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-847f7f5f44-wqqzc"]
Sep 18 20:21:39.982684 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:39.982616    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:39.985530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:39.985495    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9gb3eaktc0u5t\""
Sep 18 20:21:39.986458 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:39.986436    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 20:21:39.986582 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:39.986521    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 20:21:39.986582 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:39.986557    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2fh9b\""
Sep 18 20:21:39.986687 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:39.986576    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 20:21:39.986750 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:39.986693    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 20:21:40.072324 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.072287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-secret-metrics-server-client-certs\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.072324 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.072334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlrsg\" (UniqueName: \"kubernetes.io/projected/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-kube-api-access-mlrsg\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.072570 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.072384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.072570 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.072428    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-metrics-server-audit-profiles\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.072570 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.072538    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-audit-log\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.072570 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.072557    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-secret-metrics-server-tls\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.072732 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.072608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-client-ca-bundle\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.173786 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.173743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-secret-metrics-server-client-certs\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.173979 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.173801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlrsg\" (UniqueName: \"kubernetes.io/projected/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-kube-api-access-mlrsg\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.173979 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.173854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.173979 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.173883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-metrics-server-audit-profiles\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.174146 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.174010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-audit-log\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.174146 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.174039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-secret-metrics-server-tls\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.174146 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.174138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-client-ca-bundle\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.174731 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.174683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-audit-log\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.175011 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.174947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.175555 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.175532    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-metrics-server-audit-profiles\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.177030 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.177011    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-secret-metrics-server-client-certs\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.177502 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.177481    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-client-ca-bundle\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.177600 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.177513    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-secret-metrics-server-tls\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.187076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.187046    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlrsg\" (UniqueName: \"kubernetes.io/projected/a82816f1-006f-4fe6-bc8b-e63beb86b5b5-kube-api-access-mlrsg\") pod \"metrics-server-847f7f5f44-wqqzc\" (UID: \"a82816f1-006f-4fe6-bc8b-e63beb86b5b5\") " pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.295029 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.294946    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:21:40.446514 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.446481    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"]
Sep 18 20:21:40.491736 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.491702    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"]
Sep 18 20:21:40.491736 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.491741    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c4b498c7c-9lntn"]
Sep 18 20:21:40.491943 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.491831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"
Sep 18 20:21:40.494437 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.494413    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 20:21:40.494556 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.494485    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-r7wht\""
Sep 18 20:21:40.526104 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.526078    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4b498c7c-9lntn"]
Sep 18 20:21:40.526218 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.526118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.577337 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.577268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-trusted-ca-bundle\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.577337 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.577315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-oauth-config\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.577522 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.577348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-config\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.577522 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.577378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-service-ca\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.577522 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.577408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f6f1e772-aee4-4faa-bf29-c5bd39845eb4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-q5w68\" (UID: \"f6f1e772-aee4-4faa-bf29-c5bd39845eb4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"
Sep 18 20:21:40.577522 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.577506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm8r5\" (UniqueName: \"kubernetes.io/projected/eec20a9a-5706-486d-bc85-cc7e2200fb19-kube-api-access-mm8r5\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.577747 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.577552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-serving-cert\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.577747 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.577578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q48cm\" (UniqueName: \"kubernetes.io/projected/f6f1e772-aee4-4faa-bf29-c5bd39845eb4-kube-api-access-q48cm\") pod \"monitoring-plugin-5984459ff8-q5w68\" (UID: \"f6f1e772-aee4-4faa-bf29-c5bd39845eb4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"
Sep 18 20:21:40.577747 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.577632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-oauth-serving-cert\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.678160 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.678090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-oauth-serving-cert\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.678294 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.678207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-trusted-ca-bundle\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.678294 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.678248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-oauth-config\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.678407 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.678387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-config\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.678457 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.678442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-service-ca\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.678496 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.678481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f6f1e772-aee4-4faa-bf29-c5bd39845eb4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-q5w68\" (UID: \"f6f1e772-aee4-4faa-bf29-c5bd39845eb4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"
Sep 18 20:21:40.678527 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.678516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8r5\" (UniqueName: \"kubernetes.io/projected/eec20a9a-5706-486d-bc85-cc7e2200fb19-kube-api-access-mm8r5\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.678577 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.678555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-serving-cert\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.678621 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.678579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q48cm\" (UniqueName: \"kubernetes.io/projected/f6f1e772-aee4-4faa-bf29-c5bd39845eb4-kube-api-access-q48cm\") pod \"monitoring-plugin-5984459ff8-q5w68\" (UID: \"f6f1e772-aee4-4faa-bf29-c5bd39845eb4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"
Sep 18 20:21:40.678864 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:40.678670    2583 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 18 20:21:40.678864 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:21:40.678756    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6f1e772-aee4-4faa-bf29-c5bd39845eb4-monitoring-plugin-cert podName:f6f1e772-aee4-4faa-bf29-c5bd39845eb4 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:41.178734779 +0000 UTC m=+52.086637042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/f6f1e772-aee4-4faa-bf29-c5bd39845eb4-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-q5w68" (UID: "f6f1e772-aee4-4faa-bf29-c5bd39845eb4") : secret "monitoring-plugin-cert" not found
Sep 18 20:21:40.679074 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.678917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-oauth-serving-cert\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.679074 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.679060    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-config\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.679197 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.679090    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-service-ca\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.679197 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.679181    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-trusted-ca-bundle\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.680815 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.680794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-oauth-config\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.681213 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.681193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-serving-cert\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.687700 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.687676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q48cm\" (UniqueName: \"kubernetes.io/projected/f6f1e772-aee4-4faa-bf29-c5bd39845eb4-kube-api-access-q48cm\") pod \"monitoring-plugin-5984459ff8-q5w68\" (UID: \"f6f1e772-aee4-4faa-bf29-c5bd39845eb4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"
Sep 18 20:21:40.687765 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.687732    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8r5\" (UniqueName: \"kubernetes.io/projected/eec20a9a-5706-486d-bc85-cc7e2200fb19-kube-api-access-mm8r5\") pod \"console-6c4b498c7c-9lntn\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") " pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.834291 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.834049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:40.841341 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.841313    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"]
Sep 18 20:21:40.851237 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.851214    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.854305 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.854268    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 20:21:40.854791 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.854379    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 20:21:40.854791 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.854394    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 20:21:40.854791 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.854384    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6xjn5\""
Sep 18 20:21:40.854791 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.854534    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 20:21:40.854948 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.854807    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 20:21:40.859773 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.859273    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"]
Sep 18 20:21:40.861012 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.860995    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 20:21:40.880420 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.880388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.880548 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.880439    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-metrics-client-ca\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.880548 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.880497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-telemeter-client-tls\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.880725 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.880567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.880725 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.880600    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-secret-telemeter-client\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.880725 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.880716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-federate-client-tls\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.880866 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.880753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-serving-certs-ca-bundle\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.880866 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.880788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtlh\" (UniqueName: \"kubernetes.io/projected/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-kube-api-access-frtlh\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.981859 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.981823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-secret-telemeter-client\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.982349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.981921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-federate-client-tls\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.982349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.981951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-serving-certs-ca-bundle\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.982349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.981995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtlh\" (UniqueName: \"kubernetes.io/projected/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-kube-api-access-frtlh\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.982349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.982069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.982349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.982099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-metrics-client-ca\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.982349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.982166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-telemeter-client-tls\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.982349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.982223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.982816 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.982793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-serving-certs-ca-bundle\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.982958 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.982937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.983321 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.983288    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-metrics-client-ca\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.985095 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.985047    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.985260 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.985234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-federate-client-tls\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.985978 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.985917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-telemeter-client-tls\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.986564 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.986532    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-secret-telemeter-client\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:40.991447 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:40.991428    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtlh\" (UniqueName: \"kubernetes.io/projected/f2cd9e5c-88b5-4c16-882d-4fe793b9d02c-kube-api-access-frtlh\") pod \"telemeter-client-6ff99f5d4b-rgqq9\" (UID: \"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c\") " pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:41.164108 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:41.164068    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"
Sep 18 20:21:41.184358 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:41.184326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f6f1e772-aee4-4faa-bf29-c5bd39845eb4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-q5w68\" (UID: \"f6f1e772-aee4-4faa-bf29-c5bd39845eb4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"
Sep 18 20:21:41.186893 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:41.186872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f6f1e772-aee4-4faa-bf29-c5bd39845eb4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-q5w68\" (UID: \"f6f1e772-aee4-4faa-bf29-c5bd39845eb4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"
Sep 18 20:21:41.401493 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:41.401383    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"
Sep 18 20:21:42.571057 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:42.571024    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gp7m9" event={"ID":"d7ae1a48-dec2-4ae8-9e44-02402f265518","Type":"ContainerStarted","Data":"d74672527044c32b496bf0112168d3d6e640ffc99007766524060d9ef3033800"}
Sep 18 20:21:42.941370 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:42.941188    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85d85db47d-29rrf"]
Sep 18 20:21:42.942344 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:21:42.942323    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ed5f4a_9853_4d69_872f_cfb6074afecc.slice/crio-d23ae348bfc78defe2680c11375657ef1058825ec3dda4d54c6de98bb4a58007 WatchSource:0}: Error finding container d23ae348bfc78defe2680c11375657ef1058825ec3dda4d54c6de98bb4a58007: Status 404 returned error can't find the container with id d23ae348bfc78defe2680c11375657ef1058825ec3dda4d54c6de98bb4a58007
Sep 18 20:21:43.022586 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.022245    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:21:43.031355 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.030964    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-847f7f5f44-wqqzc"]
Sep 18 20:21:43.051276 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.044989    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9"]
Sep 18 20:21:43.063881 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:21:43.063843    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cd9e5c_88b5_4c16_882d_4fe793b9d02c.slice/crio-b370214b0744b1081edc9a406c3a94c4a4ba9e276e2e0c450bdbfe9c09e79ed8 WatchSource:0}: Error finding container b370214b0744b1081edc9a406c3a94c4a4ba9e276e2e0c450bdbfe9c09e79ed8: Status 404 returned error can't find the container with id b370214b0744b1081edc9a406c3a94c4a4ba9e276e2e0c450bdbfe9c09e79ed8
Sep 18 20:21:43.082627 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.082588    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"]
Sep 18 20:21:43.087698 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.087629    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4b498c7c-9lntn"]
Sep 18 20:21:43.587161 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.586895    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw" event={"ID":"9531a7cb-4020-4d03-bde2-df7edfb474ec","Type":"ContainerStarted","Data":"60dc53f8045d161d11552f8dab9655efc8b9efc36f9d70dd8f2459e912aa861e"}
Sep 18 20:21:43.588072 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.587179    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw" event={"ID":"9531a7cb-4020-4d03-bde2-df7edfb474ec","Type":"ContainerStarted","Data":"99c7bece8a3d7649f7e4b389f4456c5cafea6c6ff4c057ec3a43164868600816"}
Sep 18 20:21:43.588072 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.587202    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw" event={"ID":"9531a7cb-4020-4d03-bde2-df7edfb474ec","Type":"ContainerStarted","Data":"b62eba86da4592ef183b18e11df730d4af13a9a2de8205031e58406d66cf2183"}
Sep 18 20:21:43.588532 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.588498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68" event={"ID":"f6f1e772-aee4-4faa-bf29-c5bd39845eb4","Type":"ContainerStarted","Data":"97cb76f86bce2bab5227199735c743a35429b7b16aac9ed7e9eff076432ac0e7"}
Sep 18 20:21:43.591417 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.591391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dfc68" event={"ID":"a824cb57-ceee-40c5-927e-d3c6bbc13df4","Type":"ContainerStarted","Data":"035d26afbd809d3a1921053b58b6e4823f3026dff20b3d7e2408357d31fee182"}
Sep 18 20:21:43.592810 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.592789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9" event={"ID":"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c","Type":"ContainerStarted","Data":"b370214b0744b1081edc9a406c3a94c4a4ba9e276e2e0c450bdbfe9c09e79ed8"}
Sep 18 20:21:43.594463 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.594442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerStarted","Data":"7c22d0db085b1a54da98d0fbc6f2f76b1972f343ca6bea575dc667aefce7b2ad"}
Sep 18 20:21:43.597407 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.597377    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7cd4bd84-pjnmf" event={"ID":"cfe139a5-1a94-4614-aedd-34016aa65250","Type":"ContainerStarted","Data":"27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a"}
Sep 18 20:21:43.599353 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.599322    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-plv2v" event={"ID":"9ddba8b7-bff2-4785-9db8-ddd7c810c350","Type":"ContainerStarted","Data":"2a49182313df1e3e54a9e7afdc67c32f89255c432c14f61fd42150826215e3d5"}
Sep 18 20:21:43.604586 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.604564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k6v7r" event={"ID":"3e375926-9dc6-49d2-865b-fa57f024b925","Type":"ContainerStarted","Data":"9be08b47fc3a8cc2ebb995b01502c86e76595b228e50d2a1d10ca66882fa62ef"}
Sep 18 20:21:43.604691 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.604639    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k6v7r" event={"ID":"3e375926-9dc6-49d2-865b-fa57f024b925","Type":"ContainerStarted","Data":"dcfd6b077714d21a06f98d1ae420f0aa7cfceba64db08bc558e8b18c5e4c879d"}
Sep 18 20:21:43.604775 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.604693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:43.606547 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.606504    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-6cdhw" podStartSLOduration=3.066750719 podStartE2EDuration="8.606487627s" totalImagesPullingTime="5.539736908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:43.605152086 +0000 UTC m=+54.513054367" watchObservedRunningTime="2026-09-18 20:21:43.606487627 +0000 UTC m=+54.514389904"
Sep 18 20:21:43.606861 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.606535    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf" event={"ID":"f1ed5f4a-9853-4d69-872f-cfb6074afecc","Type":"ContainerStarted","Data":"d23ae348bfc78defe2680c11375657ef1058825ec3dda4d54c6de98bb4a58007"}
Sep 18 20:21:43.609010 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.608993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6" event={"ID":"00bf8807-5f69-4594-b985-f46b1423b94c","Type":"ContainerStarted","Data":"3ea65872aea584b310fa6fbad1896063fb7974f5981f6690c94f4f13379204a9"}
Sep 18 20:21:43.611626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.611602    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4b498c7c-9lntn" event={"ID":"eec20a9a-5706-486d-bc85-cc7e2200fb19","Type":"ContainerStarted","Data":"31130adfb6f803f9b1100314da4b3249d7c30325d0b2a502ed84ff93dc7e94f9"}
Sep 18 20:21:43.611773 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.611756    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4b498c7c-9lntn" event={"ID":"eec20a9a-5706-486d-bc85-cc7e2200fb19","Type":"ContainerStarted","Data":"c4054bccabd2aca1cc7857d6208a19b1b0c1b03048247fa2420de5a85c6bbfc9"}
Sep 18 20:21:43.615891 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.615866    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc" event={"ID":"a82816f1-006f-4fe6-bc8b-e63beb86b5b5","Type":"ContainerStarted","Data":"2a4148d39be84e4a5ab77dfe68dae4dcd961dac7023d2f6d70538bbaff098d08"}
Sep 18 20:21:43.622478 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.621605    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-plv2v" podStartSLOduration=1.605939408 podStartE2EDuration="9.621587728s" totalImagesPullingTime="8.01564832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:43.620789727 +0000 UTC m=+54.528692012" watchObservedRunningTime="2026-09-18 20:21:43.621587728 +0000 UTC m=+54.529490007"
Sep 18 20:21:43.645331 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.645283    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k6v7r" podStartSLOduration=4.551762904 podStartE2EDuration="9.645263911s" totalImagesPullingTime="5.093501007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:43.644485748 +0000 UTC m=+54.552388044" watchObservedRunningTime="2026-09-18 20:21:43.645263911 +0000 UTC m=+54.553166191"
Sep 18 20:21:43.665086 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.664466    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dfc68" podStartSLOduration=3.374925183 podStartE2EDuration="9.664445765s" totalImagesPullingTime="6.289520582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:21:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:43.663306075 +0000 UTC m=+54.571208355" watchObservedRunningTime="2026-09-18 20:21:43.664445765 +0000 UTC m=+54.572348046"
Sep 18 20:21:43.683503 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.682680    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f7cd4bd84-pjnmf" podStartSLOduration=1.732607413 podStartE2EDuration="9.682664808s" totalImagesPullingTime="7.950057395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:43.681338597 +0000 UTC m=+54.589240875" watchObservedRunningTime="2026-09-18 20:21:43.682664808 +0000 UTC m=+54.590567089"
Sep 18 20:21:43.705146 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:43.704668    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c4b498c7c-9lntn" podStartSLOduration=3.704637968 podStartE2EDuration="3.704637968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:43.70332538 +0000 UTC m=+54.611227660" watchObservedRunningTime="2026-09-18 20:21:43.704637968 +0000 UTC m=+54.612540248"
Sep 18 20:21:44.524969 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:44.524864    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:44.524969 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:44.524932    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:44.533268 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:44.532986    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:44.630772 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:44.630591    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:21:48.640163 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.640060    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf" event={"ID":"f1ed5f4a-9853-4d69-872f-cfb6074afecc","Type":"ContainerStarted","Data":"2cf1d6a01b741864fd2564d62a3030b41161cc25c76d9c75c925821deb087484"}
Sep 18 20:21:48.640542 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.640203    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf" event={"ID":"f1ed5f4a-9853-4d69-872f-cfb6074afecc","Type":"ContainerStarted","Data":"4a9e2765bd3dbc8a42a8a532154d28be68d4774511c122e8c7f22f33703c3293"}
Sep 18 20:21:48.640542 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.640250    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf" event={"ID":"f1ed5f4a-9853-4d69-872f-cfb6074afecc","Type":"ContainerStarted","Data":"300e8b8d54cfb71a66a750e5c968014c7e2c495057188f9017589c36ffe60d6a"}
Sep 18 20:21:48.642019 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.641991    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6" event={"ID":"00bf8807-5f69-4594-b985-f46b1423b94c","Type":"ContainerStarted","Data":"e07164a2b94f329a4f7d416153532e7db228cdd90892ee5a785ca3d8ed2ee5c6"}
Sep 18 20:21:48.643539 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.643512    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc" event={"ID":"a82816f1-006f-4fe6-bc8b-e63beb86b5b5","Type":"ContainerStarted","Data":"f1cc09e101ceaa8278df9c61f35607e3d2bbce999d92ab686227245c9fef6883"}
Sep 18 20:21:48.644998 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.644975    2583 generic.go:356] "Generic (PLEG): container finished" podID="d7ae1a48-dec2-4ae8-9e44-02402f265518" containerID="58218760f38cc50b1808800879de34ac86e283f8395acb80358e003d0ce41e3d" exitCode=0
Sep 18 20:21:48.645094 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.645058    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gp7m9" event={"ID":"d7ae1a48-dec2-4ae8-9e44-02402f265518","Type":"ContainerDied","Data":"58218760f38cc50b1808800879de34ac86e283f8395acb80358e003d0ce41e3d"}
Sep 18 20:21:48.646623 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.646604    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68" event={"ID":"f6f1e772-aee4-4faa-bf29-c5bd39845eb4","Type":"ContainerStarted","Data":"9fea10df4b7e85933ba5fc58c6083558d8e3e1495e3066f1e1cadd7ad33af246"}
Sep 18 20:21:48.646758 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.646734    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"
Sep 18 20:21:48.648990 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.648968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9" event={"ID":"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c","Type":"ContainerStarted","Data":"27106950dd3e401e9606ba549ff9dad44f0a0cfb848464de50222e2425ab9266"}
Sep 18 20:21:48.649090 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.649016    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9" event={"ID":"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c","Type":"ContainerStarted","Data":"0823562ab64e6d80f1e2b1e7eecb9b764c6c0e130ec44a2f61f37658cd5fd309"}
Sep 18 20:21:48.649090 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.649045    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9" event={"ID":"f2cd9e5c-88b5-4c16-882d-4fe793b9d02c","Type":"ContainerStarted","Data":"77d9a7508a87c9ceec77a7714a364317a31695d81771367d6ed1585ae28c5a3c"}
Sep 18 20:21:48.650522 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.650502    2583 generic.go:356] "Generic (PLEG): container finished" podID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerID="b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a" exitCode=0
Sep 18 20:21:48.650596 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.650533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerDied","Data":"b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a"}
Sep 18 20:21:48.652026 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.652006    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68"
Sep 18 20:21:48.663527 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.663485    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pw4h6" podStartSLOduration=8.713510963000001 podStartE2EDuration="13.66347196s" totalImagesPullingTime="4.949960997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:48.661407648 +0000 UTC m=+59.569309930" watchObservedRunningTime="2026-09-18 20:21:48.66347196 +0000 UTC m=+59.571374236"
Sep 18 20:21:48.678651 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.678603    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q5w68" podStartSLOduration=3.935193439 podStartE2EDuration="8.678589063s" totalImagesPullingTime="4.743395624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:48.677460296 +0000 UTC m=+59.585362574" watchObservedRunningTime="2026-09-18 20:21:48.678589063 +0000 UTC m=+59.586491332"
Sep 18 20:21:48.717619 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.717562    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc" podStartSLOduration=4.920128655 podStartE2EDuration="9.717541355s" totalImagesPullingTime="4.7974127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:48.715536703 +0000 UTC m=+59.623438993" watchObservedRunningTime="2026-09-18 20:21:48.717541355 +0000 UTC m=+59.625443636"
Sep 18 20:21:48.739363 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:48.739322    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6ff99f5d4b-rgqq9" podStartSLOduration=3.965841916 podStartE2EDuration="8.739304823s" totalImagesPullingTime="4.773462907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:48.73741375 +0000 UTC m=+59.645316033" watchObservedRunningTime="2026-09-18 20:21:48.739304823 +0000 UTC m=+59.647207096"
Sep 18 20:21:49.022429 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.022392    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4b498c7c-9lntn"]
Sep 18 20:21:49.052658 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.052618    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cd84b789-qfgk9"]
Sep 18 20:21:49.054990 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.054968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.066359 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.066339    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd84b789-qfgk9"]
Sep 18 20:21:49.167055 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.167021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-oauth-config\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.167196 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.167078    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-service-ca\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.167196 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.167165    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx97s\" (UniqueName: \"kubernetes.io/projected/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-kube-api-access-hx97s\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.167304 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.167223    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-serving-cert\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.167304 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.167248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-oauth-serving-cert\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.167304 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.167295    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-trusted-ca-bundle\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.167439 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.167333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-config\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.268482 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.268456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-oauth-config\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.268605 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.268487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-service-ca\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.268605 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.268513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx97s\" (UniqueName: \"kubernetes.io/projected/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-kube-api-access-hx97s\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.268605 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.268539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-serving-cert\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.268605 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.268561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-oauth-serving-cert\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.268605 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.268595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-trusted-ca-bundle\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.268873 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.268622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-config\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.269360 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.269330    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-service-ca\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.269677 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.269650    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-oauth-serving-cert\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.269839 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.269816    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-trusted-ca-bundle\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.269882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.269824    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-config\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.271392 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.271284    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-oauth-config\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.271483 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.271392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-serving-cert\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.277247 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.277195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx97s\" (UniqueName: \"kubernetes.io/projected/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-kube-api-access-hx97s\") pod \"console-cd84b789-qfgk9\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") " pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.364434 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.364405    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:49.644351 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.644283    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd84b789-qfgk9"]
Sep 18 20:21:49.647796 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:21:49.647767    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd090ebd7_f9a6_4264_b5dc_360b0f2e1e7a.slice/crio-d1fead0778c8b6b5613bd700a29057ca467e908197dfb75fe52c8b47d6e2fcb4 WatchSource:0}: Error finding container d1fead0778c8b6b5613bd700a29057ca467e908197dfb75fe52c8b47d6e2fcb4: Status 404 returned error can't find the container with id d1fead0778c8b6b5613bd700a29057ca467e908197dfb75fe52c8b47d6e2fcb4
Sep 18 20:21:49.657426 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.657351    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gp7m9" event={"ID":"d7ae1a48-dec2-4ae8-9e44-02402f265518","Type":"ContainerStarted","Data":"22e9734c09cff23b6bd97148070abde3c2600dcb2d74172885c627e45f79adb2"}
Sep 18 20:21:49.657426 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.657399    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gp7m9" event={"ID":"d7ae1a48-dec2-4ae8-9e44-02402f265518","Type":"ContainerStarted","Data":"e50fbde96f9c2e3aa3d09303d782784d01570ea9ba80bc54dde32af4a5bc5c57"}
Sep 18 20:21:49.658850 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.658822    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd84b789-qfgk9" event={"ID":"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a","Type":"ContainerStarted","Data":"d1fead0778c8b6b5613bd700a29057ca467e908197dfb75fe52c8b47d6e2fcb4"}
Sep 18 20:21:49.662294 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.662269    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf" event={"ID":"f1ed5f4a-9853-4d69-872f-cfb6074afecc","Type":"ContainerStarted","Data":"25579488b2a1c68390c7cd6017e4a87e140ea220897ac2baef691ee2ee3b77d6"}
Sep 18 20:21:49.662384 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.662338    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf" event={"ID":"f1ed5f4a-9853-4d69-872f-cfb6074afecc","Type":"ContainerStarted","Data":"d2de502c0b51a9a2aaab9eaff160d30716cdc9ab80e30f9d557ede9fb7ca39f9"}
Sep 18 20:21:49.663050 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.663032    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:49.663146 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.663094    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf" event={"ID":"f1ed5f4a-9853-4d69-872f-cfb6074afecc","Type":"ContainerStarted","Data":"8340d0b18d44a4217b924f923f67d8358f0c5a5130e6ec5cc67ab2ecd0f10ffe"}
Sep 18 20:21:49.677715 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:49.677663    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gp7m9" podStartSLOduration=10.443569536 podStartE2EDuration="14.677649405s" totalImagesPullingTime="4.153155535s" totalInitContainerRuntime="80.924334ms" isStatefulPod=true podCreationTimestamp="2026-09-18 20:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:49.676778066 +0000 UTC m=+60.584680346" watchObservedRunningTime="2026-09-18 20:21:49.677649405 +0000 UTC m=+60.585551683"
Sep 18 20:21:50.471644 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:50.471624    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4xrg"
Sep 18 20:21:50.559442 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:50.559387    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf" podStartSLOduration=7.709804538 podStartE2EDuration="13.559364283s" totalImagesPullingTime="5.849559745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:49.698206858 +0000 UTC m=+60.606109138" watchObservedRunningTime="2026-09-18 20:21:50.559364283 +0000 UTC m=+61.467266564"
Sep 18 20:21:50.669594 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:50.669557    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerStarted","Data":"a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a"}
Sep 18 20:21:50.670006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:50.669665    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerStarted","Data":"44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359"}
Sep 18 20:21:50.670006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:50.669709    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerStarted","Data":"9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1"}
Sep 18 20:21:50.670006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:50.669754    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerStarted","Data":"21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711"}
Sep 18 20:21:50.670006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:50.669793    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerStarted","Data":"697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286"}
Sep 18 20:21:50.670006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:50.669831    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerStarted","Data":"e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a"}
Sep 18 20:21:50.671268 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:50.671212    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd84b789-qfgk9" event={"ID":"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a","Type":"ContainerStarted","Data":"52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900"}
Sep 18 20:21:50.713791 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:50.713737    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.281014152000001 podStartE2EDuration="14.713721456s" totalImagesPullingTime="6.406500193s" totalInitContainerRuntime="26.207111ms" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:50.71293816 +0000 UTC m=+61.620840440" watchObservedRunningTime="2026-09-18 20:21:50.713721456 +0000 UTC m=+61.621623788"
Sep 18 20:21:50.834692 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:50.834622    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:21:53.626304 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:53.626274    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k6v7r"
Sep 18 20:21:53.644012 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:53.643971    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cd84b789-qfgk9" podStartSLOduration=4.643957415 podStartE2EDuration="4.643957415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:50.749308795 +0000 UTC m=+61.657211077" watchObservedRunningTime="2026-09-18 20:21:53.643957415 +0000 UTC m=+64.551859692"
Sep 18 20:21:56.680509 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:56.680480    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-85d85db47d-29rrf"
Sep 18 20:21:57.046759 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.046655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:57.046759 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.046722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:57.049625 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.049602    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 20:21:57.049733 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.049648    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 20:21:57.060156 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.060110    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/941899a6-9517-410e-b439-94c967f6c162-original-pull-secret\") pod \"global-pull-secret-syncer-9rj8d\" (UID: \"941899a6-9517-410e-b439-94c967f6c162\") " pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:57.060255 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.060203    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf22891-e55a-433a-a689-fb5b3e3575f9-metrics-certs\") pod \"network-metrics-daemon-wkf7p\" (UID: \"eaf22891-e55a-433a-a689-fb5b3e3575f9\") " pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:57.134493 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.134460    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ls8xl\""
Sep 18 20:21:57.141892 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.141868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkf7p"
Sep 18 20:21:57.141892 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.141889    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rj8d"
Sep 18 20:21:57.148845 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.148824    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pp4l\" (UniqueName: \"kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l\") pod \"network-check-target-qvkkm\" (UID: \"1ecf386e-6ab3-4448-9c76-75a96b3cfa75\") " pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:57.151550 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.151529    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 20:21:57.162177 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.162158    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 20:21:57.173380 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.173350    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pp4l\" (UniqueName: \"kubernetes.io/projected/1ecf386e-6ab3-4448-9c76-75a96b3cfa75-kube-api-access-6pp4l\") pod \"network-check-target-qvkkm\" (UID: \"1ecf386e-6ab3-4448-9c76-75a96b3cfa75\") " pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:57.425084 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.425000    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9rj8d"]
Sep 18 20:21:57.426397 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.426372    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wkf7p"]
Sep 18 20:21:57.427266 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:21:57.427244    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941899a6_9517_410e_b439_94c967f6c162.slice/crio-935a9aa4ec148df87f8d2f56bf46d8b9e8e8c69b1a0c760c85b7e1edf68d3484 WatchSource:0}: Error finding container 935a9aa4ec148df87f8d2f56bf46d8b9e8e8c69b1a0c760c85b7e1edf68d3484: Status 404 returned error can't find the container with id 935a9aa4ec148df87f8d2f56bf46d8b9e8e8c69b1a0c760c85b7e1edf68d3484
Sep 18 20:21:57.428008 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:21:57.427986    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf22891_e55a_433a_a689_fb5b3e3575f9.slice/crio-4ac4e3202a74a6f634806cdb6b62b11f8077842eed8130c4a55f63024f1bef10 WatchSource:0}: Error finding container 4ac4e3202a74a6f634806cdb6b62b11f8077842eed8130c4a55f63024f1bef10: Status 404 returned error can't find the container with id 4ac4e3202a74a6f634806cdb6b62b11f8077842eed8130c4a55f63024f1bef10
Sep 18 20:21:57.440369 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.440349    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5lg4r\""
Sep 18 20:21:57.448797 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.448774    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:21:57.699805 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.699725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wkf7p" event={"ID":"eaf22891-e55a-433a-a689-fb5b3e3575f9","Type":"ContainerStarted","Data":"4ac4e3202a74a6f634806cdb6b62b11f8077842eed8130c4a55f63024f1bef10"}
Sep 18 20:21:57.701147 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.701104    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9rj8d" event={"ID":"941899a6-9517-410e-b439-94c967f6c162","Type":"ContainerStarted","Data":"935a9aa4ec148df87f8d2f56bf46d8b9e8e8c69b1a0c760c85b7e1edf68d3484"}
Sep 18 20:21:57.720632 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:57.720605    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvkkm"]
Sep 18 20:21:57.724419 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:21:57.724388    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecf386e_6ab3_4448_9c76_75a96b3cfa75.slice/crio-f7b16aed967cc4cb6d4f530d20f590d9aff646a29f80de78c0f6167db40f5cb0 WatchSource:0}: Error finding container f7b16aed967cc4cb6d4f530d20f590d9aff646a29f80de78c0f6167db40f5cb0: Status 404 returned error can't find the container with id f7b16aed967cc4cb6d4f530d20f590d9aff646a29f80de78c0f6167db40f5cb0
Sep 18 20:21:58.708208 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:58.708174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wkf7p" event={"ID":"eaf22891-e55a-433a-a689-fb5b3e3575f9","Type":"ContainerStarted","Data":"1f1510be0adb370314058fff0baa42990b0477a5b4a127d57312cbd859ec6c0b"}
Sep 18 20:21:58.710056 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:58.710015    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvkkm" event={"ID":"1ecf386e-6ab3-4448-9c76-75a96b3cfa75","Type":"ContainerStarted","Data":"f7b16aed967cc4cb6d4f530d20f590d9aff646a29f80de78c0f6167db40f5cb0"}
Sep 18 20:21:59.364784 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:59.364746    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:59.364982 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:59.364824    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:59.371362 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:59.371333    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:59.718512 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:59.717982    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wkf7p" event={"ID":"eaf22891-e55a-433a-a689-fb5b3e3575f9","Type":"ContainerStarted","Data":"7879594dbf057172b13333fcc232a06cf3ba957e644b2aea680048b33d16e9a2"}
Sep 18 20:21:59.723495 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:59.723472    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:21:59.741930 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:59.741882    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wkf7p" podStartSLOduration=67.681888935 podStartE2EDuration="1m8.741861882s" totalImagesPullingTime="1.059972947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:59.741374052 +0000 UTC m=+70.649276331" watchObservedRunningTime="2026-09-18 20:21:59.741861882 +0000 UTC m=+70.649764162"
Sep 18 20:21:59.794658 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:21:59.794620    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f7cd4bd84-pjnmf"]
Sep 18 20:22:00.296141 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:00.296082    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:22:00.296359 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:00.296207    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:22:02.731804 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:02.731766    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9rj8d" event={"ID":"941899a6-9517-410e-b439-94c967f6c162","Type":"ContainerStarted","Data":"5b4dbd7425af097c7aeb681eef960694b1d4d8299615dffb26d500e7b1f2658b"}
Sep 18 20:22:02.733350 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:02.733327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvkkm" event={"ID":"1ecf386e-6ab3-4448-9c76-75a96b3cfa75","Type":"ContainerStarted","Data":"75a363799ca658ab8359ea02d0668c0b0fd5769032dd1372326f595a906c20f1"}
Sep 18 20:22:02.733481 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:02.733466    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:22:02.748451 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:02.748404    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9rj8d" podStartSLOduration=67.08489415 podStartE2EDuration="1m11.748387309s" totalImagesPullingTime="4.663493159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:02.747111026 +0000 UTC m=+73.655013305" watchObservedRunningTime="2026-09-18 20:22:02.748387309 +0000 UTC m=+73.656289589"
Sep 18 20:22:02.762773 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:02.762735    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qvkkm" podStartSLOduration=67.399697733 podStartE2EDuration="1m11.762722826s" totalImagesPullingTime="4.363025093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:02.761818024 +0000 UTC m=+73.669720313" watchObservedRunningTime="2026-09-18 20:22:02.762722826 +0000 UTC m=+73.670625116"
Sep 18 20:22:14.044958 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:14.044892    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c4b498c7c-9lntn" podUID="eec20a9a-5706-486d-bc85-cc7e2200fb19" containerName="console" containerID="cri-o://31130adfb6f803f9b1100314da4b3249d7c30325d0b2a502ed84ff93dc7e94f9" gracePeriod=15
Sep 18 20:22:14.780506 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:14.780466    2583 generic.go:356] "Generic (PLEG): container finished" podID="eec20a9a-5706-486d-bc85-cc7e2200fb19" containerID="31130adfb6f803f9b1100314da4b3249d7c30325d0b2a502ed84ff93dc7e94f9" exitCode=0
Sep 18 20:22:14.780628 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:14.780533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4b498c7c-9lntn" event={"ID":"eec20a9a-5706-486d-bc85-cc7e2200fb19","Type":"ContainerDied","Data":"31130adfb6f803f9b1100314da4b3249d7c30325d0b2a502ed84ff93dc7e94f9"}
Sep 18 20:22:14.854749 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:14.854727    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:22:15.012305 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012227    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-oauth-serving-cert\") pod \"eec20a9a-5706-486d-bc85-cc7e2200fb19\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") "
Sep 18 20:22:15.012305 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012278    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-serving-cert\") pod \"eec20a9a-5706-486d-bc85-cc7e2200fb19\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") "
Sep 18 20:22:15.012305 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012303    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-config\") pod \"eec20a9a-5706-486d-bc85-cc7e2200fb19\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") "
Sep 18 20:22:15.012584 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012417    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-trusted-ca-bundle\") pod \"eec20a9a-5706-486d-bc85-cc7e2200fb19\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") "
Sep 18 20:22:15.012584 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012445    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-service-ca\") pod \"eec20a9a-5706-486d-bc85-cc7e2200fb19\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") "
Sep 18 20:22:15.012584 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012472    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-oauth-config\") pod \"eec20a9a-5706-486d-bc85-cc7e2200fb19\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") "
Sep 18 20:22:15.012584 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012513    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm8r5\" (UniqueName: \"kubernetes.io/projected/eec20a9a-5706-486d-bc85-cc7e2200fb19-kube-api-access-mm8r5\") pod \"eec20a9a-5706-486d-bc85-cc7e2200fb19\" (UID: \"eec20a9a-5706-486d-bc85-cc7e2200fb19\") "
Sep 18 20:22:15.012779 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012670    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eec20a9a-5706-486d-bc85-cc7e2200fb19" (UID: "eec20a9a-5706-486d-bc85-cc7e2200fb19"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:15.012779 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012743    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-config" (OuterVolumeSpecName: "console-config") pod "eec20a9a-5706-486d-bc85-cc7e2200fb19" (UID: "eec20a9a-5706-486d-bc85-cc7e2200fb19"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:15.012886 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012791    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-service-ca" (OuterVolumeSpecName: "service-ca") pod "eec20a9a-5706-486d-bc85-cc7e2200fb19" (UID: "eec20a9a-5706-486d-bc85-cc7e2200fb19"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:15.012886 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012834    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "eec20a9a-5706-486d-bc85-cc7e2200fb19" (UID: "eec20a9a-5706-486d-bc85-cc7e2200fb19"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:15.012886 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012869    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-oauth-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.012886 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012886    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.013059 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.012896    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-service-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.014896 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.014860    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec20a9a-5706-486d-bc85-cc7e2200fb19-kube-api-access-mm8r5" (OuterVolumeSpecName: "kube-api-access-mm8r5") pod "eec20a9a-5706-486d-bc85-cc7e2200fb19" (UID: "eec20a9a-5706-486d-bc85-cc7e2200fb19"). InnerVolumeSpecName "kube-api-access-mm8r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:15.014896 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.014876    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eec20a9a-5706-486d-bc85-cc7e2200fb19" (UID: "eec20a9a-5706-486d-bc85-cc7e2200fb19"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:15.015036 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.014912    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eec20a9a-5706-486d-bc85-cc7e2200fb19" (UID: "eec20a9a-5706-486d-bc85-cc7e2200fb19"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:15.113503 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.113465    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec20a9a-5706-486d-bc85-cc7e2200fb19-trusted-ca-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.113503 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.113496    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-oauth-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.113503 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.113505    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm8r5\" (UniqueName: \"kubernetes.io/projected/eec20a9a-5706-486d-bc85-cc7e2200fb19-kube-api-access-mm8r5\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.113921 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.113514    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eec20a9a-5706-486d-bc85-cc7e2200fb19-console-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.785535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.785508    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4b498c7c-9lntn"
Sep 18 20:22:15.785711 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.785508    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4b498c7c-9lntn" event={"ID":"eec20a9a-5706-486d-bc85-cc7e2200fb19","Type":"ContainerDied","Data":"c4054bccabd2aca1cc7857d6208a19b1b0c1b03048247fa2420de5a85c6bbfc9"}
Sep 18 20:22:15.785711 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.785644    2583 scope.go:117] "RemoveContainer" containerID="31130adfb6f803f9b1100314da4b3249d7c30325d0b2a502ed84ff93dc7e94f9"
Sep 18 20:22:15.801960 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.801935    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4b498c7c-9lntn"]
Sep 18 20:22:15.808558 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:15.808534    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c4b498c7c-9lntn"]
Sep 18 20:22:17.314849 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:17.314814    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eec20a9a-5706-486d-bc85-cc7e2200fb19" path="/var/lib/kubelet/pods/eec20a9a-5706-486d-bc85-cc7e2200fb19/volumes"
Sep 18 20:22:20.300814 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:20.300784    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:22:20.304492 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:20.304472    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-847f7f5f44-wqqzc"
Sep 18 20:22:24.825027 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:24.824989    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f7cd4bd84-pjnmf" podUID="cfe139a5-1a94-4614-aedd-34016aa65250" containerName="console" containerID="cri-o://27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a" gracePeriod=15
Sep 18 20:22:25.059946 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.059923    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:22:25.097311 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.097233    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-console-config\") pod \"cfe139a5-1a94-4614-aedd-34016aa65250\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") "
Sep 18 20:22:25.097311 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.097302    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfe139a5-1a94-4614-aedd-34016aa65250-console-serving-cert\") pod \"cfe139a5-1a94-4614-aedd-34016aa65250\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") "
Sep 18 20:22:25.097512 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.097357    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-trusted-ca-bundle\") pod \"cfe139a5-1a94-4614-aedd-34016aa65250\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") "
Sep 18 20:22:25.097512 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.097410    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46t5n\" (UniqueName: \"kubernetes.io/projected/cfe139a5-1a94-4614-aedd-34016aa65250-kube-api-access-46t5n\") pod \"cfe139a5-1a94-4614-aedd-34016aa65250\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") "
Sep 18 20:22:25.097512 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.097455    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-oauth-serving-cert\") pod \"cfe139a5-1a94-4614-aedd-34016aa65250\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") "
Sep 18 20:22:25.097512 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.097481    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfe139a5-1a94-4614-aedd-34016aa65250-console-oauth-config\") pod \"cfe139a5-1a94-4614-aedd-34016aa65250\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") "
Sep 18 20:22:25.097512 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.097507    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-service-ca\") pod \"cfe139a5-1a94-4614-aedd-34016aa65250\" (UID: \"cfe139a5-1a94-4614-aedd-34016aa65250\") "
Sep 18 20:22:25.097756 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.097695    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-console-config" (OuterVolumeSpecName: "console-config") pod "cfe139a5-1a94-4614-aedd-34016aa65250" (UID: "cfe139a5-1a94-4614-aedd-34016aa65250"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:25.097814 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.097787    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cfe139a5-1a94-4614-aedd-34016aa65250" (UID: "cfe139a5-1a94-4614-aedd-34016aa65250"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:25.098050 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.098012    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cfe139a5-1a94-4614-aedd-34016aa65250" (UID: "cfe139a5-1a94-4614-aedd-34016aa65250"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:25.098050 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.098029    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-service-ca" (OuterVolumeSpecName: "service-ca") pod "cfe139a5-1a94-4614-aedd-34016aa65250" (UID: "cfe139a5-1a94-4614-aedd-34016aa65250"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:25.100058 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.100032    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe139a5-1a94-4614-aedd-34016aa65250-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cfe139a5-1a94-4614-aedd-34016aa65250" (UID: "cfe139a5-1a94-4614-aedd-34016aa65250"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:25.100178 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.100151    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe139a5-1a94-4614-aedd-34016aa65250-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cfe139a5-1a94-4614-aedd-34016aa65250" (UID: "cfe139a5-1a94-4614-aedd-34016aa65250"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:25.100229 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.100187    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe139a5-1a94-4614-aedd-34016aa65250-kube-api-access-46t5n" (OuterVolumeSpecName: "kube-api-access-46t5n") pod "cfe139a5-1a94-4614-aedd-34016aa65250" (UID: "cfe139a5-1a94-4614-aedd-34016aa65250"). InnerVolumeSpecName "kube-api-access-46t5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:25.198480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.198426    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-trusted-ca-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:25.198480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.198470    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46t5n\" (UniqueName: \"kubernetes.io/projected/cfe139a5-1a94-4614-aedd-34016aa65250-kube-api-access-46t5n\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:25.198480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.198481    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-oauth-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:25.198480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.198489    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfe139a5-1a94-4614-aedd-34016aa65250-console-oauth-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:25.198480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.198501    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-service-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:25.198775 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.198509    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfe139a5-1a94-4614-aedd-34016aa65250-console-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:25.198775 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.198517    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfe139a5-1a94-4614-aedd-34016aa65250-console-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:25.828118 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.828081    2583 generic.go:356] "Generic (PLEG): container finished" podID="cfe139a5-1a94-4614-aedd-34016aa65250" containerID="27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a" exitCode=0
Sep 18 20:22:25.828570 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.828142    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7cd4bd84-pjnmf"
Sep 18 20:22:25.828570 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.828172    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7cd4bd84-pjnmf" event={"ID":"cfe139a5-1a94-4614-aedd-34016aa65250","Type":"ContainerDied","Data":"27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a"}
Sep 18 20:22:25.828570 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.828220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7cd4bd84-pjnmf" event={"ID":"cfe139a5-1a94-4614-aedd-34016aa65250","Type":"ContainerDied","Data":"2df22ca6f9726566090e70dc3a8dfbe73a8a9b09e1b216feb14cd35bd94e277f"}
Sep 18 20:22:25.828570 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.828239    2583 scope.go:117] "RemoveContainer" containerID="27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a"
Sep 18 20:22:25.836333 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.836316    2583 scope.go:117] "RemoveContainer" containerID="27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a"
Sep 18 20:22:25.836588 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:22:25.836567    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a\": container with ID starting with 27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a not found: ID does not exist" containerID="27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a"
Sep 18 20:22:25.836680 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.836595    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a"} err="failed to get container status \"27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a\": rpc error: code = NotFound desc = could not find container \"27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a\": container with ID starting with 27b99361f373493d79b2c183b5245049e740e7a0a79a7340eb51902965aa471a not found: ID does not exist"
Sep 18 20:22:25.844952 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.844930    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f7cd4bd84-pjnmf"]
Sep 18 20:22:25.848013 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:25.847995    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f7cd4bd84-pjnmf"]
Sep 18 20:22:27.316162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:27.316101    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfe139a5-1a94-4614-aedd-34016aa65250" path="/var/lib/kubelet/pods/cfe139a5-1a94-4614-aedd-34016aa65250/volumes"
Sep 18 20:22:33.739144 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:33.739092    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qvkkm"
Sep 18 20:22:55.216232 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.216195    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56f9795bbb-nczwr"]
Sep 18 20:22:55.216663 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.216492    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec20a9a-5706-486d-bc85-cc7e2200fb19" containerName="console"
Sep 18 20:22:55.216663 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.216503    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec20a9a-5706-486d-bc85-cc7e2200fb19" containerName="console"
Sep 18 20:22:55.216663 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.216522    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe139a5-1a94-4614-aedd-34016aa65250" containerName="console"
Sep 18 20:22:55.216663 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.216527    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe139a5-1a94-4614-aedd-34016aa65250" containerName="console"
Sep 18 20:22:55.216663 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.216585    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe139a5-1a94-4614-aedd-34016aa65250" containerName="console"
Sep 18 20:22:55.216663 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.216592    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec20a9a-5706-486d-bc85-cc7e2200fb19" containerName="console"
Sep 18 20:22:55.220538 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.220517    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.231593 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.231571    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f9795bbb-nczwr"]
Sep 18 20:22:55.347480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.347454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-config\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.347638 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.347487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-oauth-config\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.347638 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.347512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-trusted-ca-bundle\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.347638 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.347602    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-service-ca\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.347638 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.347632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-oauth-serving-cert\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.347769 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.347652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94nc6\" (UniqueName: \"kubernetes.io/projected/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-kube-api-access-94nc6\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.347769 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.347672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-serving-cert\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.448392 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.448356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-config\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.448392 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.448394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-oauth-config\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.448578 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.448425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-trusted-ca-bundle\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.448578 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.448470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-service-ca\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.448578 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.448494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-oauth-serving-cert\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.448578 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.448519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94nc6\" (UniqueName: \"kubernetes.io/projected/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-kube-api-access-94nc6\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.448578 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.448544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-serving-cert\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.449296 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.449274    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-config\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.449418 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.449316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-oauth-serving-cert\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.449418 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.449316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-service-ca\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.449418 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.449369    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-trusted-ca-bundle\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.451030 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.451005    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-serving-cert\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.451144 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.451108    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-oauth-config\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.458003 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.457982    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94nc6\" (UniqueName: \"kubernetes.io/projected/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-kube-api-access-94nc6\") pod \"console-56f9795bbb-nczwr\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") " pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.530157 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.530061    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:22:55.799463 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.799436    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f9795bbb-nczwr"]
Sep 18 20:22:55.801868 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:22:55.801841    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77e2af6_50bf_484f_97fa_6dd2fdfc04db.slice/crio-d548c91f073c2d2032dd7e9be693b4298f2943f711363d30135604503fbf0d20 WatchSource:0}: Error finding container d548c91f073c2d2032dd7e9be693b4298f2943f711363d30135604503fbf0d20: Status 404 returned error can't find the container with id d548c91f073c2d2032dd7e9be693b4298f2943f711363d30135604503fbf0d20
Sep 18 20:22:55.945845 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.945812    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f9795bbb-nczwr" event={"ID":"f77e2af6-50bf-484f-97fa-6dd2fdfc04db","Type":"ContainerStarted","Data":"7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1"}
Sep 18 20:22:55.946038 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.945865    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f9795bbb-nczwr" event={"ID":"f77e2af6-50bf-484f-97fa-6dd2fdfc04db","Type":"ContainerStarted","Data":"d548c91f073c2d2032dd7e9be693b4298f2943f711363d30135604503fbf0d20"}
Sep 18 20:22:55.963818 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.963771    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56f9795bbb-nczwr" podStartSLOduration=0.963756081 podStartE2EDuration="963.756081ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:55.962778288 +0000 UTC m=+126.870680579" watchObservedRunningTime="2026-09-18 20:22:55.963756081 +0000 UTC m=+126.871658359"
Sep 18 20:22:55.993305 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.993268    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:22:55.993894 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.993858    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="alertmanager" containerID="cri-o://e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a" gracePeriod=120
Sep 18 20:22:55.994014 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.993939    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="kube-rbac-proxy-web" containerID="cri-o://21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711" gracePeriod=120
Sep 18 20:22:55.994014 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.993972    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="prom-label-proxy" containerID="cri-o://a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a" gracePeriod=120
Sep 18 20:22:55.994014 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.993963    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="config-reloader" containerID="cri-o://697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286" gracePeriod=120
Sep 18 20:22:55.994224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.993924    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="kube-rbac-proxy-metric" containerID="cri-o://44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359" gracePeriod=120
Sep 18 20:22:55.994224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:55.994057    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="kube-rbac-proxy" containerID="cri-o://9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1" gracePeriod=120
Sep 18 20:22:56.953417 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:56.953388    2583 generic.go:356] "Generic (PLEG): container finished" podID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerID="a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a" exitCode=0
Sep 18 20:22:56.953417 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:56.953412    2583 generic.go:356] "Generic (PLEG): container finished" podID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerID="9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1" exitCode=0
Sep 18 20:22:56.953417 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:56.953417    2583 generic.go:356] "Generic (PLEG): container finished" podID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerID="697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286" exitCode=0
Sep 18 20:22:56.953417 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:56.953423    2583 generic.go:356] "Generic (PLEG): container finished" podID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerID="e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a" exitCode=0
Sep 18 20:22:56.953934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:56.953471    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerDied","Data":"a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a"}
Sep 18 20:22:56.953934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:56.953537    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerDied","Data":"9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1"}
Sep 18 20:22:56.953934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:56.953565    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerDied","Data":"697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286"}
Sep 18 20:22:56.953934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:56.953591    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerDied","Data":"e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a"}
Sep 18 20:22:57.229843 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.229820    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:57.367498 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367468    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-main-tls\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.367677 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367511    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ceb26837-53a9-42bc-b65c-bd96eea63443-config-out\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.367677 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367533    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.367677 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367549    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h84wf\" (UniqueName: \"kubernetes.io/projected/ceb26837-53a9-42bc-b65c-bd96eea63443-kube-api-access-h84wf\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.367846 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367682    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-web-config\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.367846 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367733    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy-web\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.367846 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367774    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-config-volume\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.367846 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367801    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb26837-53a9-42bc-b65c-bd96eea63443-alertmanager-trusted-ca-bundle\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.367846 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367842    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ceb26837-53a9-42bc-b65c-bd96eea63443-alertmanager-main-db\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.368100 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367880    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceb26837-53a9-42bc-b65c-bd96eea63443-metrics-client-ca\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.368100 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367912    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.368100 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367976    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ceb26837-53a9-42bc-b65c-bd96eea63443-tls-assets\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.368100 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.367998    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-cluster-tls-config\") pod \"ceb26837-53a9-42bc-b65c-bd96eea63443\" (UID: \"ceb26837-53a9-42bc-b65c-bd96eea63443\") "
Sep 18 20:22:57.368535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.368508    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb26837-53a9-42bc-b65c-bd96eea63443-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:57.370833 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.370799    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:57.371207 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.371179    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb26837-53a9-42bc-b65c-bd96eea63443-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:57.371207 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.371176    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:57.371382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.371241    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb26837-53a9-42bc-b65c-bd96eea63443-kube-api-access-h84wf" (OuterVolumeSpecName: "kube-api-access-h84wf") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "kube-api-access-h84wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:57.371382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.371261    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb26837-53a9-42bc-b65c-bd96eea63443-config-out" (OuterVolumeSpecName: "config-out") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:57.371382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.371263    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:57.371545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.371421    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-config-volume" (OuterVolumeSpecName: "config-volume") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:57.371545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.371486    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb26837-53a9-42bc-b65c-bd96eea63443-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:57.373057 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.373032    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:57.373363 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.373340    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb26837-53a9-42bc-b65c-bd96eea63443-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:57.374899 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.374881    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:57.381298 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.381277    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-web-config" (OuterVolumeSpecName: "web-config") pod "ceb26837-53a9-42bc-b65c-bd96eea63443" (UID: "ceb26837-53a9-42bc-b65c-bd96eea63443"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:57.468948 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.468860    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ceb26837-53a9-42bc-b65c-bd96eea63443-config-out\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.468948 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.468897    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.468948 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.468913    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h84wf\" (UniqueName: \"kubernetes.io/projected/ceb26837-53a9-42bc-b65c-bd96eea63443-kube-api-access-h84wf\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.468948 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.468927    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-web-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.468948 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.468942    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.469306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.468956    2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-config-volume\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.469306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.468970    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb26837-53a9-42bc-b65c-bd96eea63443-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.469306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.468983    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ceb26837-53a9-42bc-b65c-bd96eea63443-alertmanager-main-db\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.469306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.468997    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ceb26837-53a9-42bc-b65c-bd96eea63443-metrics-client-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.469306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.469009    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.469306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.469022    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ceb26837-53a9-42bc-b65c-bd96eea63443-tls-assets\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.469306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.469158    2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-cluster-tls-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.469306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.469186    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ceb26837-53a9-42bc-b65c-bd96eea63443-secret-alertmanager-main-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:22:57.960979 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.960949    2583 generic.go:356] "Generic (PLEG): container finished" podID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerID="44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359" exitCode=0
Sep 18 20:22:57.960979 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.960976    2583 generic.go:356] "Generic (PLEG): container finished" podID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerID="21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711" exitCode=0
Sep 18 20:22:57.961440 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.961034    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:57.961440 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.961027    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerDied","Data":"44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359"}
Sep 18 20:22:57.961440 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.961178    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerDied","Data":"21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711"}
Sep 18 20:22:57.961440 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.961207    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ceb26837-53a9-42bc-b65c-bd96eea63443","Type":"ContainerDied","Data":"7c22d0db085b1a54da98d0fbc6f2f76b1972f343ca6bea575dc667aefce7b2ad"}
Sep 18 20:22:57.961440 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.961237    2583 scope.go:117] "RemoveContainer" containerID="a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a"
Sep 18 20:22:57.969227 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.969208    2583 scope.go:117] "RemoveContainer" containerID="44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359"
Sep 18 20:22:57.976387 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.976373    2583 scope.go:117] "RemoveContainer" containerID="9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1"
Sep 18 20:22:57.982859 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.982843    2583 scope.go:117] "RemoveContainer" containerID="21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711"
Sep 18 20:22:57.985951 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.985931    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:22:57.990592 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.990568    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:22:57.991097 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.991083    2583 scope.go:117] "RemoveContainer" containerID="697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286"
Sep 18 20:22:57.998180 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:57.998160    2583 scope.go:117] "RemoveContainer" containerID="e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a"
Sep 18 20:22:58.005056 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.005038    2583 scope.go:117] "RemoveContainer" containerID="b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a"
Sep 18 20:22:58.013539 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.013523    2583 scope.go:117] "RemoveContainer" containerID="a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a"
Sep 18 20:22:58.013794 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:22:58.013776    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a\": container with ID starting with a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a not found: ID does not exist" containerID="a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a"
Sep 18 20:22:58.013853 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.013805    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a"} err="failed to get container status \"a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a\": rpc error: code = NotFound desc = could not find container \"a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a\": container with ID starting with a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a not found: ID does not exist"
Sep 18 20:22:58.013853 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.013825    2583 scope.go:117] "RemoveContainer" containerID="44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359"
Sep 18 20:22:58.014189 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:22:58.014094    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359\": container with ID starting with 44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359 not found: ID does not exist" containerID="44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359"
Sep 18 20:22:58.014189 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.014142    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359"} err="failed to get container status \"44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359\": rpc error: code = NotFound desc = could not find container \"44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359\": container with ID starting with 44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359 not found: ID does not exist"
Sep 18 20:22:58.014189 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.014165    2583 scope.go:117] "RemoveContainer" containerID="9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1"
Sep 18 20:22:58.014818 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:22:58.014787    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1\": container with ID starting with 9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1 not found: ID does not exist" containerID="9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1"
Sep 18 20:22:58.014891 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.014815    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1"} err="failed to get container status \"9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1\": rpc error: code = NotFound desc = could not find container \"9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1\": container with ID starting with 9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1 not found: ID does not exist"
Sep 18 20:22:58.014891 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.014831    2583 scope.go:117] "RemoveContainer" containerID="21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711"
Sep 18 20:22:58.015205 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:22:58.015101    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711\": container with ID starting with 21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711 not found: ID does not exist" containerID="21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711"
Sep 18 20:22:58.015205 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.015158    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711"} err="failed to get container status \"21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711\": rpc error: code = NotFound desc = could not find container \"21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711\": container with ID starting with 21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711 not found: ID does not exist"
Sep 18 20:22:58.015205 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.015186    2583 scope.go:117] "RemoveContainer" containerID="697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286"
Sep 18 20:22:58.015449 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:22:58.015435    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286\": container with ID starting with 697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286 not found: ID does not exist" containerID="697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286"
Sep 18 20:22:58.015488 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.015452    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286"} err="failed to get container status \"697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286\": rpc error: code = NotFound desc = could not find container \"697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286\": container with ID starting with 697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286 not found: ID does not exist"
Sep 18 20:22:58.015488 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.015465    2583 scope.go:117] "RemoveContainer" containerID="e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a"
Sep 18 20:22:58.015692 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:22:58.015677    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a\": container with ID starting with e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a not found: ID does not exist" containerID="e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a"
Sep 18 20:22:58.015729 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.015696    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a"} err="failed to get container status \"e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a\": rpc error: code = NotFound desc = could not find container \"e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a\": container with ID starting with e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a not found: ID does not exist"
Sep 18 20:22:58.015729 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.015708    2583 scope.go:117] "RemoveContainer" containerID="b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a"
Sep 18 20:22:58.015953 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:22:58.015930    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a\": container with ID starting with b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a not found: ID does not exist" containerID="b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a"
Sep 18 20:22:58.015999 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.015963    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a"} err="failed to get container status \"b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a\": rpc error: code = NotFound desc = could not find container \"b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a\": container with ID starting with b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a not found: ID does not exist"
Sep 18 20:22:58.015999 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.015982    2583 scope.go:117] "RemoveContainer" containerID="a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a"
Sep 18 20:22:58.016260 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016242    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a"} err="failed to get container status \"a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a\": rpc error: code = NotFound desc = could not find container \"a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a\": container with ID starting with a7c05a412a021e4ebeacad21cb841c0447ca620d9037c3a73b1fd60ef978ed5a not found: ID does not exist"
Sep 18 20:22:58.016318 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016262    2583 scope.go:117] "RemoveContainer" containerID="44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359"
Sep 18 20:22:58.016398 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016383    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:22:58.016443 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016427    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359"} err="failed to get container status \"44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359\": rpc error: code = NotFound desc = could not find container \"44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359\": container with ID starting with 44473744ee07543367033bcc60b200cc13be6f002fba4a30240a2ccd6b537359 not found: ID does not exist"
Sep 18 20:22:58.016483 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016444    2583 scope.go:117] "RemoveContainer" containerID="9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1"
Sep 18 20:22:58.016669 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016654    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1"} err="failed to get container status \"9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1\": rpc error: code = NotFound desc = could not find container \"9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1\": container with ID starting with 9ecb46f0de011d920fbdf1f6735906a27820fb5441876cd8a548eaeea446bfc1 not found: ID does not exist"
Sep 18 20:22:58.016721 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016670    2583 scope.go:117] "RemoveContainer" containerID="21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711"
Sep 18 20:22:58.016775 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016762    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="kube-rbac-proxy-metric"
Sep 18 20:22:58.016814 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016779    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="kube-rbac-proxy-metric"
Sep 18 20:22:58.016814 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016794    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="init-config-reloader"
Sep 18 20:22:58.016814 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016803    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="init-config-reloader"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016816    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="kube-rbac-proxy"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016825    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="kube-rbac-proxy"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016839    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="alertmanager"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016847    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="alertmanager"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016855    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="kube-rbac-proxy-web"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016861    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="kube-rbac-proxy-web"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016867    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="config-reloader"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016872    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="config-reloader"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016878    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="prom-label-proxy"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016885    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="prom-label-proxy"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016892    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711"} err="failed to get container status \"21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711\": rpc error: code = NotFound desc = could not find container \"21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711\": container with ID starting with 21277194aae57087222bcd1b4d93302e853a407672e96758c1a8c0b052f61711 not found: ID does not exist"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016905    2583 scope.go:117] "RemoveContainer" containerID="697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286"
Sep 18 20:22:58.016934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016936    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="alertmanager"
Sep 18 20:22:58.017376 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016943    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="kube-rbac-proxy"
Sep 18 20:22:58.017376 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016951    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="config-reloader"
Sep 18 20:22:58.017376 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016962    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="kube-rbac-proxy-metric"
Sep 18 20:22:58.017376 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016971    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="prom-label-proxy"
Sep 18 20:22:58.017376 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.016978    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" containerName="kube-rbac-proxy-web"
Sep 18 20:22:58.017376 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.017177    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286"} err="failed to get container status \"697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286\": rpc error: code = NotFound desc = could not find container \"697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286\": container with ID starting with 697c243f49ab29f14e8235deca60416aced966b3a2317af35a3e23581a8bf286 not found: ID does not exist"
Sep 18 20:22:58.017376 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.017192    2583 scope.go:117] "RemoveContainer" containerID="e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a"
Sep 18 20:22:58.017599 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.017398    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a"} err="failed to get container status \"e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a\": rpc error: code = NotFound desc = could not find container \"e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a\": container with ID starting with e0e59c8e6d00fc07fbde6971da6fe9618a2e2479c686be676c8efd496aad4d4a not found: ID does not exist"
Sep 18 20:22:58.017599 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.017410    2583 scope.go:117] "RemoveContainer" containerID="b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a"
Sep 18 20:22:58.017704 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.017669    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a"} err="failed to get container status \"b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a\": rpc error: code = NotFound desc = could not find container \"b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a\": container with ID starting with b925973a2859871e5e9932649f9633f5bb02d310efd60bfc3eb0894e70176d7a not found: ID does not exist"
Sep 18 20:22:58.020328 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.020313    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.023375 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.023355    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 20:22:58.023375 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.023365    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 20:22:58.023531 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.023398    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 20:22:58.023531 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.023415    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 20:22:58.023531 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.023432    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 20:22:58.023531 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.023355    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 20:22:58.023531 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.023505    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 20:22:58.023789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.023771    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 20:22:58.023900 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.023884    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gtw5k\""
Sep 18 20:22:58.027889 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.027874    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 20:22:58.034694 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.034675    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:22:58.174888 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.174857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c3894f6-14c5-4135-a98b-f5b78c8e809c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.174888 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.174889    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.175078 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.174909    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.175078 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.174937    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c3894f6-14c5-4135-a98b-f5b78c8e809c-config-out\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.175078 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.174978    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5nr2\" (UniqueName: \"kubernetes.io/projected/1c3894f6-14c5-4135-a98b-f5b78c8e809c-kube-api-access-w5nr2\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.175078 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.175011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c3894f6-14c5-4135-a98b-f5b78c8e809c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.175078 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.175046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.175078 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.175068    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c3894f6-14c5-4135-a98b-f5b78c8e809c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.175294 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.175113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.175294 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.175163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.175294 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.175189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.175294 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.175205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c3894f6-14c5-4135-a98b-f5b78c8e809c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.175294 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.175220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-web-config\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276312 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c3894f6-14c5-4135-a98b-f5b78c8e809c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276312 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276312 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c3894f6-14c5-4135-a98b-f5b78c8e809c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276339    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276365    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c3894f6-14c5-4135-a98b-f5b78c8e809c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-web-config\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c3894f6-14c5-4135-a98b-f5b78c8e809c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276476    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c3894f6-14c5-4135-a98b-f5b78c8e809c-config-out\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.276953 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.276568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5nr2\" (UniqueName: \"kubernetes.io/projected/1c3894f6-14c5-4135-a98b-f5b78c8e809c-kube-api-access-w5nr2\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.277715 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.277507    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c3894f6-14c5-4135-a98b-f5b78c8e809c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.277715 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.277662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c3894f6-14c5-4135-a98b-f5b78c8e809c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.277901 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.277797    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c3894f6-14c5-4135-a98b-f5b78c8e809c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.279461 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.279434    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c3894f6-14c5-4135-a98b-f5b78c8e809c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.279804 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.279500    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.279804 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.279749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.279928 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.279829    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.280469 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.280448    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-web-config\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.280532 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.280483    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.280727 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.280705    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.280764 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.280756    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c3894f6-14c5-4135-a98b-f5b78c8e809c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.281435 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.281419    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c3894f6-14c5-4135-a98b-f5b78c8e809c-config-out\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.286936 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.286916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5nr2\" (UniqueName: \"kubernetes.io/projected/1c3894f6-14c5-4135-a98b-f5b78c8e809c-kube-api-access-w5nr2\") pod \"alertmanager-main-0\" (UID: \"1c3894f6-14c5-4135-a98b-f5b78c8e809c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.329024 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.328989    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 20:22:58.607603 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.607579    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 20:22:58.609237 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:22:58.609209    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3894f6_14c5_4135_a98b_f5b78c8e809c.slice/crio-b6692808024db62cddfa783e9d3653785b704490f4a2c05491ff1983ecb8cc42 WatchSource:0}: Error finding container b6692808024db62cddfa783e9d3653785b704490f4a2c05491ff1983ecb8cc42: Status 404 returned error can't find the container with id b6692808024db62cddfa783e9d3653785b704490f4a2c05491ff1983ecb8cc42
Sep 18 20:22:58.967897 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.967865    2583 generic.go:356] "Generic (PLEG): container finished" podID="1c3894f6-14c5-4135-a98b-f5b78c8e809c" containerID="91d07c9345e601e1e336f9df769fe4fb82171bad954eb35172259a632eebd1f3" exitCode=0
Sep 18 20:22:58.968270 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.967904    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c3894f6-14c5-4135-a98b-f5b78c8e809c","Type":"ContainerDied","Data":"91d07c9345e601e1e336f9df769fe4fb82171bad954eb35172259a632eebd1f3"}
Sep 18 20:22:58.968270 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:58.967954    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c3894f6-14c5-4135-a98b-f5b78c8e809c","Type":"ContainerStarted","Data":"b6692808024db62cddfa783e9d3653785b704490f4a2c05491ff1983ecb8cc42"}
Sep 18 20:22:59.320694 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:59.320669    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb26837-53a9-42bc-b65c-bd96eea63443" path="/var/lib/kubelet/pods/ceb26837-53a9-42bc-b65c-bd96eea63443/volumes"
Sep 18 20:22:59.976179 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:59.976148    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c3894f6-14c5-4135-a98b-f5b78c8e809c","Type":"ContainerStarted","Data":"8264c48c55ca6376913fdf82e55d9721958c058a34bbe20d9a32d8ea23ece7d7"}
Sep 18 20:22:59.976555 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:59.976207    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c3894f6-14c5-4135-a98b-f5b78c8e809c","Type":"ContainerStarted","Data":"65d569e5930168873a67d0f1c033e9f9161c9ce468ffe05af89fc29c3987e34d"}
Sep 18 20:22:59.976555 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:59.976236    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c3894f6-14c5-4135-a98b-f5b78c8e809c","Type":"ContainerStarted","Data":"77d8a9ab7d34846dee0a0206d8d842adbe09f69778ca850e5e5fa2dab960f347"}
Sep 18 20:22:59.976555 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:59.976261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c3894f6-14c5-4135-a98b-f5b78c8e809c","Type":"ContainerStarted","Data":"bdfd4514a9111cb2b859ea49bf03c060fcd91c7a39cc236fd073acb881d96ef1"}
Sep 18 20:22:59.976555 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:59.976284    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c3894f6-14c5-4135-a98b-f5b78c8e809c","Type":"ContainerStarted","Data":"2307e4e60192b43740529bf7bf79cb7ed65b5122e5abe979572cf8334838b0b7"}
Sep 18 20:22:59.976555 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:22:59.976306    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c3894f6-14c5-4135-a98b-f5b78c8e809c","Type":"ContainerStarted","Data":"4e612747a19fa6fbe902c65975a07e000fd2cc774443d5b3cf0daba36c7437bf"}
Sep 18 20:23:00.008079 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:00.007996    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9925458259999997 podStartE2EDuration="3.007974506s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.42868ms" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:00.006358322 +0000 UTC m=+130.914260600" watchObservedRunningTime="2026-09-18 20:23:00.007974506 +0000 UTC m=+130.915876786"
Sep 18 20:23:05.530980 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:05.530947    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:23:05.531366 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:05.531007    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:23:05.535761 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:05.535738    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:23:06.004228 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:06.004198    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:23:06.048489 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:06.048453    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd84b789-qfgk9"]
Sep 18 20:23:31.071478 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.071352    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cd84b789-qfgk9" podUID="d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" containerName="console" containerID="cri-o://52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900" gracePeriod=15
Sep 18 20:23:31.312563 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.312529    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:23:31.444613 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.444586    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-config\") pod \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") "
Sep 18 20:23:31.444768 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.444619    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-trusted-ca-bundle\") pod \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") "
Sep 18 20:23:31.444768 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.444649    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-oauth-serving-cert\") pod \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") "
Sep 18 20:23:31.444842 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.444787    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-service-ca\") pod \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") "
Sep 18 20:23:31.444877 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.444845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-oauth-config\") pod \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") "
Sep 18 20:23:31.444913 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.444898    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-serving-cert\") pod \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") "
Sep 18 20:23:31.444955 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.444935    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx97s\" (UniqueName: \"kubernetes.io/projected/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-kube-api-access-hx97s\") pod \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\" (UID: \"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a\") "
Sep 18 20:23:31.445065 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.445029    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-config" (OuterVolumeSpecName: "console-config") pod "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" (UID: "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:31.445065 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.445037    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" (UID: "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:31.445211 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.445056    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" (UID: "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:31.445211 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.445179    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-service-ca" (OuterVolumeSpecName: "service-ca") pod "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" (UID: "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:31.445310 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.445242    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-oauth-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:23:31.445310 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.445263    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-service-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:23:31.445310 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.445275    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:23:31.445310 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.445287    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-trusted-ca-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:23:31.447287 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.447256    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" (UID: "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:31.447384 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.447313    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-kube-api-access-hx97s" (OuterVolumeSpecName: "kube-api-access-hx97s") pod "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" (UID: "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a"). InnerVolumeSpecName "kube-api-access-hx97s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:31.447384 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.447333    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" (UID: "d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:31.546588 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.546541    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-oauth-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:23:31.546588 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.546582    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-console-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:23:31.546588 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:31.546592    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx97s\" (UniqueName: \"kubernetes.io/projected/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a-kube-api-access-hx97s\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:23:32.101633 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:32.101596    2583 generic.go:356] "Generic (PLEG): container finished" podID="d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" containerID="52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900" exitCode=0
Sep 18 20:23:32.102105 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:32.101659    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd84b789-qfgk9" event={"ID":"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a","Type":"ContainerDied","Data":"52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900"}
Sep 18 20:23:32.102105 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:32.101671    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd84b789-qfgk9"
Sep 18 20:23:32.102105 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:32.101724    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd84b789-qfgk9" event={"ID":"d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a","Type":"ContainerDied","Data":"d1fead0778c8b6b5613bd700a29057ca467e908197dfb75fe52c8b47d6e2fcb4"}
Sep 18 20:23:32.102105 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:32.101744    2583 scope.go:117] "RemoveContainer" containerID="52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900"
Sep 18 20:23:32.110656 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:32.110637    2583 scope.go:117] "RemoveContainer" containerID="52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900"
Sep 18 20:23:32.110913 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:23:32.110896    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900\": container with ID starting with 52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900 not found: ID does not exist" containerID="52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900"
Sep 18 20:23:32.110947 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:32.110922    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900"} err="failed to get container status \"52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900\": rpc error: code = NotFound desc = could not find container \"52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900\": container with ID starting with 52f762087d0f62ef605f3b2f7407ea5348b20cb96fc1db61ae594dadf479b900 not found: ID does not exist"
Sep 18 20:23:32.122008 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:32.121984    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd84b789-qfgk9"]
Sep 18 20:23:32.126298 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:32.126278    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cd84b789-qfgk9"]
Sep 18 20:23:33.314943 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:23:33.314904    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" path="/var/lib/kubelet/pods/d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a/volumes"
Sep 18 20:24:32.117389 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.117351    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"]
Sep 18 20:24:32.117853 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.117674    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" containerName="console"
Sep 18 20:24:32.117853 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.117685    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" containerName="console"
Sep 18 20:24:32.117853 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.117740    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d090ebd7-f9a6-4264-b5dc-360b0f2e1e7a" containerName="console"
Sep 18 20:24:32.120733 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.120719    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:32.123701 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.123681    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 20:24:32.123701 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.123693    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 20:24:32.124503 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.124476    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jhlfx\""
Sep 18 20:24:32.129424 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.129402    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"]
Sep 18 20:24:32.219727 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.219691    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f775879f-6168-4e9d-80ba-3f5a6e65e360-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7\" (UID: \"f775879f-6168-4e9d-80ba-3f5a6e65e360\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:32.219902 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.219749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f775879f-6168-4e9d-80ba-3f5a6e65e360-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7\" (UID: \"f775879f-6168-4e9d-80ba-3f5a6e65e360\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:32.219902 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.219773    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28nxd\" (UniqueName: \"kubernetes.io/projected/f775879f-6168-4e9d-80ba-3f5a6e65e360-kube-api-access-28nxd\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7\" (UID: \"f775879f-6168-4e9d-80ba-3f5a6e65e360\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:32.320402 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.320375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f775879f-6168-4e9d-80ba-3f5a6e65e360-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7\" (UID: \"f775879f-6168-4e9d-80ba-3f5a6e65e360\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:32.320542 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.320410    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28nxd\" (UniqueName: \"kubernetes.io/projected/f775879f-6168-4e9d-80ba-3f5a6e65e360-kube-api-access-28nxd\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7\" (UID: \"f775879f-6168-4e9d-80ba-3f5a6e65e360\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:32.320542 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.320463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f775879f-6168-4e9d-80ba-3f5a6e65e360-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7\" (UID: \"f775879f-6168-4e9d-80ba-3f5a6e65e360\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:32.320831 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.320810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f775879f-6168-4e9d-80ba-3f5a6e65e360-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7\" (UID: \"f775879f-6168-4e9d-80ba-3f5a6e65e360\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:32.320869 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.320822    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f775879f-6168-4e9d-80ba-3f5a6e65e360-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7\" (UID: \"f775879f-6168-4e9d-80ba-3f5a6e65e360\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:32.330070 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.330048    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28nxd\" (UniqueName: \"kubernetes.io/projected/f775879f-6168-4e9d-80ba-3f5a6e65e360-kube-api-access-28nxd\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7\" (UID: \"f775879f-6168-4e9d-80ba-3f5a6e65e360\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:32.430566 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.430547    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:32.701518 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:32.701492    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"]
Sep 18 20:24:32.703408 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:24:32.703380    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf775879f_6168_4e9d_80ba_3f5a6e65e360.slice/crio-1f6b219999740f23bc77b05af88183c9debbf4933d56dc58870a2a9ceb959481 WatchSource:0}: Error finding container 1f6b219999740f23bc77b05af88183c9debbf4933d56dc58870a2a9ceb959481: Status 404 returned error can't find the container with id 1f6b219999740f23bc77b05af88183c9debbf4933d56dc58870a2a9ceb959481
Sep 18 20:24:33.332579 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:33.332530    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7" event={"ID":"f775879f-6168-4e9d-80ba-3f5a6e65e360","Type":"ContainerStarted","Data":"1f6b219999740f23bc77b05af88183c9debbf4933d56dc58870a2a9ceb959481"}
Sep 18 20:24:39.358996 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:39.358962    2583 generic.go:356] "Generic (PLEG): container finished" podID="f775879f-6168-4e9d-80ba-3f5a6e65e360" containerID="a8f373697b655e9cc505077aa2f1bb788db8bde27a3c5d3c408296b8c63bb144" exitCode=0
Sep 18 20:24:39.359526 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:39.359050    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7" event={"ID":"f775879f-6168-4e9d-80ba-3f5a6e65e360","Type":"ContainerDied","Data":"a8f373697b655e9cc505077aa2f1bb788db8bde27a3c5d3c408296b8c63bb144"}
Sep 18 20:24:42.372319 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:42.372282    2583 generic.go:356] "Generic (PLEG): container finished" podID="f775879f-6168-4e9d-80ba-3f5a6e65e360" containerID="997f5067ba47d9e777710aa22487af2711d097e66cc6508adf268d238621f52a" exitCode=0
Sep 18 20:24:42.372710 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:42.372365    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7" event={"ID":"f775879f-6168-4e9d-80ba-3f5a6e65e360","Type":"ContainerDied","Data":"997f5067ba47d9e777710aa22487af2711d097e66cc6508adf268d238621f52a"}
Sep 18 20:24:49.404651 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:49.404616    2583 generic.go:356] "Generic (PLEG): container finished" podID="f775879f-6168-4e9d-80ba-3f5a6e65e360" containerID="6a0459c436967434dbe7e1ef21abab8b8053e8f51d6aa651f1d4392ce605e7b7" exitCode=0
Sep 18 20:24:49.405021 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:49.404705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7" event={"ID":"f775879f-6168-4e9d-80ba-3f5a6e65e360","Type":"ContainerDied","Data":"6a0459c436967434dbe7e1ef21abab8b8053e8f51d6aa651f1d4392ce605e7b7"}
Sep 18 20:24:50.532544 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:50.532522    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:50.683949 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:50.683919    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f775879f-6168-4e9d-80ba-3f5a6e65e360-util\") pod \"f775879f-6168-4e9d-80ba-3f5a6e65e360\" (UID: \"f775879f-6168-4e9d-80ba-3f5a6e65e360\") "
Sep 18 20:24:50.684146 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:50.684002    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28nxd\" (UniqueName: \"kubernetes.io/projected/f775879f-6168-4e9d-80ba-3f5a6e65e360-kube-api-access-28nxd\") pod \"f775879f-6168-4e9d-80ba-3f5a6e65e360\" (UID: \"f775879f-6168-4e9d-80ba-3f5a6e65e360\") "
Sep 18 20:24:50.684146 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:50.684050    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f775879f-6168-4e9d-80ba-3f5a6e65e360-bundle\") pod \"f775879f-6168-4e9d-80ba-3f5a6e65e360\" (UID: \"f775879f-6168-4e9d-80ba-3f5a6e65e360\") "
Sep 18 20:24:50.684699 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:50.684667    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f775879f-6168-4e9d-80ba-3f5a6e65e360-bundle" (OuterVolumeSpecName: "bundle") pod "f775879f-6168-4e9d-80ba-3f5a6e65e360" (UID: "f775879f-6168-4e9d-80ba-3f5a6e65e360"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:50.686324 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:50.686291    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f775879f-6168-4e9d-80ba-3f5a6e65e360-kube-api-access-28nxd" (OuterVolumeSpecName: "kube-api-access-28nxd") pod "f775879f-6168-4e9d-80ba-3f5a6e65e360" (UID: "f775879f-6168-4e9d-80ba-3f5a6e65e360"). InnerVolumeSpecName "kube-api-access-28nxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:24:50.688794 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:50.688766    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f775879f-6168-4e9d-80ba-3f5a6e65e360-util" (OuterVolumeSpecName: "util") pod "f775879f-6168-4e9d-80ba-3f5a6e65e360" (UID: "f775879f-6168-4e9d-80ba-3f5a6e65e360"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:50.784959 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:50.784912    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28nxd\" (UniqueName: \"kubernetes.io/projected/f775879f-6168-4e9d-80ba-3f5a6e65e360-kube-api-access-28nxd\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:24:50.784959 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:50.784953    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f775879f-6168-4e9d-80ba-3f5a6e65e360-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:24:50.784959 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:50.784967    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f775879f-6168-4e9d-80ba-3f5a6e65e360-util\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:24:51.414259 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:51.414220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7" event={"ID":"f775879f-6168-4e9d-80ba-3f5a6e65e360","Type":"ContainerDied","Data":"1f6b219999740f23bc77b05af88183c9debbf4933d56dc58870a2a9ceb959481"}
Sep 18 20:24:51.414421 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:51.414265    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4lpq7"
Sep 18 20:24:51.414421 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:51.414270    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f6b219999740f23bc77b05af88183c9debbf4933d56dc58870a2a9ceb959481"
Sep 18 20:24:54.814027 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.813993    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"]
Sep 18 20:24:54.814416 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.814307    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f775879f-6168-4e9d-80ba-3f5a6e65e360" containerName="extract"
Sep 18 20:24:54.814416 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.814318    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f775879f-6168-4e9d-80ba-3f5a6e65e360" containerName="extract"
Sep 18 20:24:54.814416 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.814330    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f775879f-6168-4e9d-80ba-3f5a6e65e360" containerName="util"
Sep 18 20:24:54.814416 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.814335    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f775879f-6168-4e9d-80ba-3f5a6e65e360" containerName="util"
Sep 18 20:24:54.814416 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.814343    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f775879f-6168-4e9d-80ba-3f5a6e65e360" containerName="pull"
Sep 18 20:24:54.814416 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.814348    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f775879f-6168-4e9d-80ba-3f5a6e65e360" containerName="pull"
Sep 18 20:24:54.814416 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.814397    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f775879f-6168-4e9d-80ba-3f5a6e65e360" containerName="extract"
Sep 18 20:24:54.816604 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.816587    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"
Sep 18 20:24:54.819603 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.819583    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 18 20:24:54.819713 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.819607    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 20:24:54.819713 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.819668    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 20:24:54.819881 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.819868    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-2jzf4\""
Sep 18 20:24:54.832174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.832153    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"]
Sep 18 20:24:54.921412 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.921376    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2wwr\" (UniqueName: \"kubernetes.io/projected/179ea335-42ed-4161-b797-c942805ae578-kube-api-access-n2wwr\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-dzt9c\" (UID: \"179ea335-42ed-4161-b797-c942805ae578\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"
Sep 18 20:24:54.921559 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:54.921420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/179ea335-42ed-4161-b797-c942805ae578-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-dzt9c\" (UID: \"179ea335-42ed-4161-b797-c942805ae578\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"
Sep 18 20:24:55.022290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:55.022260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2wwr\" (UniqueName: \"kubernetes.io/projected/179ea335-42ed-4161-b797-c942805ae578-kube-api-access-n2wwr\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-dzt9c\" (UID: \"179ea335-42ed-4161-b797-c942805ae578\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"
Sep 18 20:24:55.022451 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:55.022296    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/179ea335-42ed-4161-b797-c942805ae578-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-dzt9c\" (UID: \"179ea335-42ed-4161-b797-c942805ae578\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"
Sep 18 20:24:55.024730 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:55.024699    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/179ea335-42ed-4161-b797-c942805ae578-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-dzt9c\" (UID: \"179ea335-42ed-4161-b797-c942805ae578\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"
Sep 18 20:24:55.033850 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:55.033827    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2wwr\" (UniqueName: \"kubernetes.io/projected/179ea335-42ed-4161-b797-c942805ae578-kube-api-access-n2wwr\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-dzt9c\" (UID: \"179ea335-42ed-4161-b797-c942805ae578\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"
Sep 18 20:24:55.127033 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:55.126945    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"
Sep 18 20:24:55.403947 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:55.403921    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"]
Sep 18 20:24:55.406311 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:24:55.406282    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179ea335_42ed_4161_b797_c942805ae578.slice/crio-d3b82451a38c9b9118cabde1a831110eb02ad7b8098e5a0e785b773e154bcc7b WatchSource:0}: Error finding container d3b82451a38c9b9118cabde1a831110eb02ad7b8098e5a0e785b773e154bcc7b: Status 404 returned error can't find the container with id d3b82451a38c9b9118cabde1a831110eb02ad7b8098e5a0e785b773e154bcc7b
Sep 18 20:24:55.430686 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:55.430657    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c" event={"ID":"179ea335-42ed-4161-b797-c942805ae578","Type":"ContainerStarted","Data":"d3b82451a38c9b9118cabde1a831110eb02ad7b8098e5a0e785b773e154bcc7b"}
Sep 18 20:24:59.110960 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.110930    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5pswm"]
Sep 18 20:24:59.112954 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.112939    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:24:59.115956 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.115939    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 18 20:24:59.116181 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.116165    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-qhdkj\""
Sep 18 20:24:59.117367 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.117354    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 20:24:59.140304 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.140283    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5pswm"]
Sep 18 20:24:59.260238 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.260209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd6ld\" (UniqueName: \"kubernetes.io/projected/3453a745-dfe9-417f-ad98-24330e4560d7-kube-api-access-rd6ld\") pod \"keda-operator-579c8958f7-5pswm\" (UID: \"3453a745-dfe9-417f-ad98-24330e4560d7\") " pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:24:59.260238 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.260256    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3453a745-dfe9-417f-ad98-24330e4560d7-cabundle0\") pod \"keda-operator-579c8958f7-5pswm\" (UID: \"3453a745-dfe9-417f-ad98-24330e4560d7\") " pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:24:59.260444 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.260324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3453a745-dfe9-417f-ad98-24330e4560d7-certificates\") pod \"keda-operator-579c8958f7-5pswm\" (UID: \"3453a745-dfe9-417f-ad98-24330e4560d7\") " pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:24:59.314955 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.314922    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"]
Sep 18 20:24:59.317012 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.316995    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:24:59.320021 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.320001    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 18 20:24:59.324758 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.324359    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"]
Sep 18 20:24:59.361068 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.361007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd6ld\" (UniqueName: \"kubernetes.io/projected/3453a745-dfe9-417f-ad98-24330e4560d7-kube-api-access-rd6ld\") pod \"keda-operator-579c8958f7-5pswm\" (UID: \"3453a745-dfe9-417f-ad98-24330e4560d7\") " pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:24:59.361068 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.361058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3453a745-dfe9-417f-ad98-24330e4560d7-cabundle0\") pod \"keda-operator-579c8958f7-5pswm\" (UID: \"3453a745-dfe9-417f-ad98-24330e4560d7\") " pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:24:59.361272 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.361172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3453a745-dfe9-417f-ad98-24330e4560d7-certificates\") pod \"keda-operator-579c8958f7-5pswm\" (UID: \"3453a745-dfe9-417f-ad98-24330e4560d7\") " pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:24:59.361360 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:24:59.361341    2583 secret.go:281] references non-existent secret key: ca.crt
Sep 18 20:24:59.361424 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:24:59.361368    2583 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 18 20:24:59.361424 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:24:59.361381    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-5pswm: references non-existent secret key: ca.crt
Sep 18 20:24:59.361520 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:24:59.361457    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3453a745-dfe9-417f-ad98-24330e4560d7-certificates podName:3453a745-dfe9-417f-ad98-24330e4560d7 nodeName:}" failed. No retries permitted until 2026-09-18 20:24:59.861436027 +0000 UTC m=+250.769338299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3453a745-dfe9-417f-ad98-24330e4560d7-certificates") pod "keda-operator-579c8958f7-5pswm" (UID: "3453a745-dfe9-417f-ad98-24330e4560d7") : references non-existent secret key: ca.crt
Sep 18 20:24:59.361665 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.361644    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3453a745-dfe9-417f-ad98-24330e4560d7-cabundle0\") pod \"keda-operator-579c8958f7-5pswm\" (UID: \"3453a745-dfe9-417f-ad98-24330e4560d7\") " pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:24:59.370770 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.370744    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd6ld\" (UniqueName: \"kubernetes.io/projected/3453a745-dfe9-417f-ad98-24330e4560d7-kube-api-access-rd6ld\") pod \"keda-operator-579c8958f7-5pswm\" (UID: \"3453a745-dfe9-417f-ad98-24330e4560d7\") " pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:24:59.451081 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.451043    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c" event={"ID":"179ea335-42ed-4161-b797-c942805ae578","Type":"ContainerStarted","Data":"6d680668235f26507a5cce6f1f6d0fc4d17d59dcfc2b636f39533d7b2bafa403"}
Sep 18 20:24:59.451250 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.451177    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"
Sep 18 20:24:59.462539 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.462514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5d62bc1-ec4d-4f71-9d46-1b272304a429-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rtv9s\" (UID: \"d5d62bc1-ec4d-4f71-9d46-1b272304a429\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:24:59.462705 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.462560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d5d62bc1-ec4d-4f71-9d46-1b272304a429-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rtv9s\" (UID: \"d5d62bc1-ec4d-4f71-9d46-1b272304a429\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:24:59.462705 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.462578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhng\" (UniqueName: \"kubernetes.io/projected/d5d62bc1-ec4d-4f71-9d46-1b272304a429-kube-api-access-zbhng\") pod \"keda-metrics-apiserver-77857b7b98-rtv9s\" (UID: \"d5d62bc1-ec4d-4f71-9d46-1b272304a429\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:24:59.473920 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.473865    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c" podStartSLOduration=2.454750024 podStartE2EDuration="5.473847438s" totalImagesPullingTime="3.019097414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:59.472355927 +0000 UTC m=+250.380258208" watchObservedRunningTime="2026-09-18 20:24:59.473847438 +0000 UTC m=+250.381749718"
Sep 18 20:24:59.563197 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.563154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5d62bc1-ec4d-4f71-9d46-1b272304a429-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rtv9s\" (UID: \"d5d62bc1-ec4d-4f71-9d46-1b272304a429\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:24:59.563373 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.563230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d5d62bc1-ec4d-4f71-9d46-1b272304a429-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rtv9s\" (UID: \"d5d62bc1-ec4d-4f71-9d46-1b272304a429\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:24:59.563373 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.563252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhng\" (UniqueName: \"kubernetes.io/projected/d5d62bc1-ec4d-4f71-9d46-1b272304a429-kube-api-access-zbhng\") pod \"keda-metrics-apiserver-77857b7b98-rtv9s\" (UID: \"d5d62bc1-ec4d-4f71-9d46-1b272304a429\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:24:59.563373 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:24:59.563291    2583 secret.go:281] references non-existent secret key: tls.crt
Sep 18 20:24:59.563373 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:24:59.563313    2583 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 18 20:24:59.563373 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:24:59.563337    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s: references non-existent secret key: tls.crt
Sep 18 20:24:59.563556 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:24:59.563418    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5d62bc1-ec4d-4f71-9d46-1b272304a429-certificates podName:d5d62bc1-ec4d-4f71-9d46-1b272304a429 nodeName:}" failed. No retries permitted until 2026-09-18 20:25:00.063399468 +0000 UTC m=+250.971301745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d5d62bc1-ec4d-4f71-9d46-1b272304a429-certificates") pod "keda-metrics-apiserver-77857b7b98-rtv9s" (UID: "d5d62bc1-ec4d-4f71-9d46-1b272304a429") : references non-existent secret key: tls.crt
Sep 18 20:24:59.563632 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.563614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d5d62bc1-ec4d-4f71-9d46-1b272304a429-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rtv9s\" (UID: \"d5d62bc1-ec4d-4f71-9d46-1b272304a429\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:24:59.574568 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.574542    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhng\" (UniqueName: \"kubernetes.io/projected/d5d62bc1-ec4d-4f71-9d46-1b272304a429-kube-api-access-zbhng\") pod \"keda-metrics-apiserver-77857b7b98-rtv9s\" (UID: \"d5d62bc1-ec4d-4f71-9d46-1b272304a429\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:24:59.705732 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.705704    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-79x8v"]
Sep 18 20:24:59.709265 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.709247    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-79x8v"
Sep 18 20:24:59.711982 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.711965    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 20:24:59.719533 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.719509    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-79x8v"]
Sep 18 20:24:59.866610 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.866577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv5r9\" (UniqueName: \"kubernetes.io/projected/ae6fe560-d83a-4289-af20-caae999ba693-kube-api-access-cv5r9\") pod \"keda-admission-6c4d864b86-79x8v\" (UID: \"ae6fe560-d83a-4289-af20-caae999ba693\") " pod="openshift-keda/keda-admission-6c4d864b86-79x8v"
Sep 18 20:24:59.866760 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.866627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae6fe560-d83a-4289-af20-caae999ba693-certificates\") pod \"keda-admission-6c4d864b86-79x8v\" (UID: \"ae6fe560-d83a-4289-af20-caae999ba693\") " pod="openshift-keda/keda-admission-6c4d864b86-79x8v"
Sep 18 20:24:59.866760 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.866680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3453a745-dfe9-417f-ad98-24330e4560d7-certificates\") pod \"keda-operator-579c8958f7-5pswm\" (UID: \"3453a745-dfe9-417f-ad98-24330e4560d7\") " pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:24:59.869363 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.869320    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3453a745-dfe9-417f-ad98-24330e4560d7-certificates\") pod \"keda-operator-579c8958f7-5pswm\" (UID: \"3453a745-dfe9-417f-ad98-24330e4560d7\") " pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:24:59.967711 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.967627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv5r9\" (UniqueName: \"kubernetes.io/projected/ae6fe560-d83a-4289-af20-caae999ba693-kube-api-access-cv5r9\") pod \"keda-admission-6c4d864b86-79x8v\" (UID: \"ae6fe560-d83a-4289-af20-caae999ba693\") " pod="openshift-keda/keda-admission-6c4d864b86-79x8v"
Sep 18 20:24:59.967711 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.967671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae6fe560-d83a-4289-af20-caae999ba693-certificates\") pod \"keda-admission-6c4d864b86-79x8v\" (UID: \"ae6fe560-d83a-4289-af20-caae999ba693\") " pod="openshift-keda/keda-admission-6c4d864b86-79x8v"
Sep 18 20:24:59.970281 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.970255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae6fe560-d83a-4289-af20-caae999ba693-certificates\") pod \"keda-admission-6c4d864b86-79x8v\" (UID: \"ae6fe560-d83a-4289-af20-caae999ba693\") " pod="openshift-keda/keda-admission-6c4d864b86-79x8v"
Sep 18 20:24:59.976426 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:24:59.976401    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv5r9\" (UniqueName: \"kubernetes.io/projected/ae6fe560-d83a-4289-af20-caae999ba693-kube-api-access-cv5r9\") pod \"keda-admission-6c4d864b86-79x8v\" (UID: \"ae6fe560-d83a-4289-af20-caae999ba693\") " pod="openshift-keda/keda-admission-6c4d864b86-79x8v"
Sep 18 20:25:00.020269 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:00.020245    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-79x8v"
Sep 18 20:25:00.022735 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:00.022712    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:25:00.068346 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:00.068311    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5d62bc1-ec4d-4f71-9d46-1b272304a429-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rtv9s\" (UID: \"d5d62bc1-ec4d-4f71-9d46-1b272304a429\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:25:00.071022 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:00.070980    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5d62bc1-ec4d-4f71-9d46-1b272304a429-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rtv9s\" (UID: \"d5d62bc1-ec4d-4f71-9d46-1b272304a429\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:25:00.229260 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:00.229150    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:25:00.301290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:00.301264    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5pswm"]
Sep 18 20:25:00.304717 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:25:00.304685    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3453a745_dfe9_417f_ad98_24330e4560d7.slice/crio-1692f160c46c2d1d8f7cb93a945db150b6c84ba3f5192cbb2c8469d98f348fd8 WatchSource:0}: Error finding container 1692f160c46c2d1d8f7cb93a945db150b6c84ba3f5192cbb2c8469d98f348fd8: Status 404 returned error can't find the container with id 1692f160c46c2d1d8f7cb93a945db150b6c84ba3f5192cbb2c8469d98f348fd8
Sep 18 20:25:00.304963 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:00.304939    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-79x8v"]
Sep 18 20:25:00.305741 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:25:00.305717    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6fe560_d83a_4289_af20_caae999ba693.slice/crio-40f91317e48b89a660047874026d04239151869e43e318e495aa1922630deae0 WatchSource:0}: Error finding container 40f91317e48b89a660047874026d04239151869e43e318e495aa1922630deae0: Status 404 returned error can't find the container with id 40f91317e48b89a660047874026d04239151869e43e318e495aa1922630deae0
Sep 18 20:25:00.456547 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:00.456511    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-79x8v" event={"ID":"ae6fe560-d83a-4289-af20-caae999ba693","Type":"ContainerStarted","Data":"40f91317e48b89a660047874026d04239151869e43e318e495aa1922630deae0"}
Sep 18 20:25:00.457784 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:00.457757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5pswm" event={"ID":"3453a745-dfe9-417f-ad98-24330e4560d7","Type":"ContainerStarted","Data":"1692f160c46c2d1d8f7cb93a945db150b6c84ba3f5192cbb2c8469d98f348fd8"}
Sep 18 20:25:00.503975 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:00.503951    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"]
Sep 18 20:25:00.506414 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:25:00.506392    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d62bc1_ec4d_4f71_9d46_1b272304a429.slice/crio-332b54f9680b6281a80282c0abd566c171f21cdf1ddcbcf71d71d68440412f05 WatchSource:0}: Error finding container 332b54f9680b6281a80282c0abd566c171f21cdf1ddcbcf71d71d68440412f05: Status 404 returned error can't find the container with id 332b54f9680b6281a80282c0abd566c171f21cdf1ddcbcf71d71d68440412f05
Sep 18 20:25:01.466441 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:01.466394    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s" event={"ID":"d5d62bc1-ec4d-4f71-9d46-1b272304a429","Type":"ContainerStarted","Data":"332b54f9680b6281a80282c0abd566c171f21cdf1ddcbcf71d71d68440412f05"}
Sep 18 20:25:02.472649 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:02.472615    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-79x8v" event={"ID":"ae6fe560-d83a-4289-af20-caae999ba693","Type":"ContainerStarted","Data":"640118dd097d06177765ea2197644f5067ffd73891ee0245e3d53cb973fd5ee7"}
Sep 18 20:25:02.473069 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:02.472703    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-79x8v"
Sep 18 20:25:02.488426 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:02.488295    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-79x8v" podStartSLOduration=2.065096259 podStartE2EDuration="3.488273924s" totalImagesPullingTime="1.423177665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:25:02.488024281 +0000 UTC m=+253.395926560" watchObservedRunningTime="2026-09-18 20:25:02.488273924 +0000 UTC m=+253.396176203"
Sep 18 20:25:04.483242 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:04.483158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5pswm" event={"ID":"3453a745-dfe9-417f-ad98-24330e4560d7","Type":"ContainerStarted","Data":"83618ca008282514648a8667e5478c481fc82e111da79f6c0971db3cb2b17b11"}
Sep 18 20:25:04.483651 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:04.483274    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:25:04.484847 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:04.484815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s" event={"ID":"d5d62bc1-ec4d-4f71-9d46-1b272304a429","Type":"ContainerStarted","Data":"a416ae63a4fca26f13a6c6cc13a58b526dcbbabe102032283c9d22324244415b"}
Sep 18 20:25:04.484941 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:04.484876    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:25:04.501438 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:04.501390    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-5pswm" podStartSLOduration=1.6263631520000001 podStartE2EDuration="5.501373836s" totalImagesPullingTime="3.875010684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:25:04.499957536 +0000 UTC m=+255.407859816" watchObservedRunningTime="2026-09-18 20:25:04.501373836 +0000 UTC m=+255.409276116"
Sep 18 20:25:04.515345 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:04.515296    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s" podStartSLOduration=1.847497057 podStartE2EDuration="5.515279324s" totalImagesPullingTime="3.667782267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:25:04.51498986 +0000 UTC m=+255.422892144" watchObservedRunningTime="2026-09-18 20:25:04.515279324 +0000 UTC m=+255.423181602"
Sep 18 20:25:15.495619 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:15.495591    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rtv9s"
Sep 18 20:25:20.460736 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:20.460705    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-dzt9c"
Sep 18 20:25:23.479892 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:23.479858    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-79x8v"
Sep 18 20:25:25.491214 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:25.491186    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-5pswm"
Sep 18 20:25:51.207585 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:51.207558    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:25:51.209391 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:51.209368    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:25:51.210989 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:25:51.210970    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 20:26:15.668454 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.668422    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-x262t"]
Sep 18 20:26:15.670647 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.670624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-x262t"
Sep 18 20:26:15.673394 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.673369    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 20:26:15.673635 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.673617    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 18 20:26:15.674676 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.674662    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-t4fjh\""
Sep 18 20:26:15.674727 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.674690    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 20:26:15.682472 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.682448    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"]
Sep 18 20:26:15.684857 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.684836    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-x262t"]
Sep 18 20:26:15.684959 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.684936    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"
Sep 18 20:26:15.687858 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.687841    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 18 20:26:15.687954 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.687848    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-htd59\""
Sep 18 20:26:15.695777 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.695750    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"]
Sep 18 20:26:15.706844 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.706813    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-ts8m6"]
Sep 18 20:26:15.710979 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.710952    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-ts8m6"
Sep 18 20:26:15.713990 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.713970    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wb7jw\""
Sep 18 20:26:15.714091 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.714004    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 20:26:15.717165 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.717141    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-ts8m6"]
Sep 18 20:26:15.809463 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.809433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3ef17b10-51f0-4100-9820-55c34ad598a5-data\") pod \"seaweedfs-64568bcd49-ts8m6\" (UID: \"3ef17b10-51f0-4100-9820-55c34ad598a5\") " pod="kserve/seaweedfs-64568bcd49-ts8m6"
Sep 18 20:26:15.809626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.809469    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3a07917-7f99-4096-9836-ab43d55b4db8-cert\") pod \"kserve-controller-manager-7dcf4bff6-x262t\" (UID: \"e3a07917-7f99-4096-9836-ab43d55b4db8\") " pod="kserve/kserve-controller-manager-7dcf4bff6-x262t"
Sep 18 20:26:15.809626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.809491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrm9c\" (UniqueName: \"kubernetes.io/projected/e3a07917-7f99-4096-9836-ab43d55b4db8-kube-api-access-xrm9c\") pod \"kserve-controller-manager-7dcf4bff6-x262t\" (UID: \"e3a07917-7f99-4096-9836-ab43d55b4db8\") " pod="kserve/kserve-controller-manager-7dcf4bff6-x262t"
Sep 18 20:26:15.809626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.809544    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd491e0b-5e5b-4c6d-a042-3d11f1fddf31-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-bhqdm\" (UID: \"dd491e0b-5e5b-4c6d-a042-3d11f1fddf31\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"
Sep 18 20:26:15.809626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.809567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2m4z\" (UniqueName: \"kubernetes.io/projected/dd491e0b-5e5b-4c6d-a042-3d11f1fddf31-kube-api-access-v2m4z\") pod \"llmisvc-controller-manager-645f9c8fd6-bhqdm\" (UID: \"dd491e0b-5e5b-4c6d-a042-3d11f1fddf31\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"
Sep 18 20:26:15.809801 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.809642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5b4k\" (UniqueName: \"kubernetes.io/projected/3ef17b10-51f0-4100-9820-55c34ad598a5-kube-api-access-h5b4k\") pod \"seaweedfs-64568bcd49-ts8m6\" (UID: \"3ef17b10-51f0-4100-9820-55c34ad598a5\") " pod="kserve/seaweedfs-64568bcd49-ts8m6"
Sep 18 20:26:15.911036 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.911006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd491e0b-5e5b-4c6d-a042-3d11f1fddf31-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-bhqdm\" (UID: \"dd491e0b-5e5b-4c6d-a042-3d11f1fddf31\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"
Sep 18 20:26:15.911229 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.911046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2m4z\" (UniqueName: \"kubernetes.io/projected/dd491e0b-5e5b-4c6d-a042-3d11f1fddf31-kube-api-access-v2m4z\") pod \"llmisvc-controller-manager-645f9c8fd6-bhqdm\" (UID: \"dd491e0b-5e5b-4c6d-a042-3d11f1fddf31\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"
Sep 18 20:26:15.911229 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.911064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5b4k\" (UniqueName: \"kubernetes.io/projected/3ef17b10-51f0-4100-9820-55c34ad598a5-kube-api-access-h5b4k\") pod \"seaweedfs-64568bcd49-ts8m6\" (UID: \"3ef17b10-51f0-4100-9820-55c34ad598a5\") " pod="kserve/seaweedfs-64568bcd49-ts8m6"
Sep 18 20:26:15.911229 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.911106    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3ef17b10-51f0-4100-9820-55c34ad598a5-data\") pod \"seaweedfs-64568bcd49-ts8m6\" (UID: \"3ef17b10-51f0-4100-9820-55c34ad598a5\") " pod="kserve/seaweedfs-64568bcd49-ts8m6"
Sep 18 20:26:15.911229 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.911146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3a07917-7f99-4096-9836-ab43d55b4db8-cert\") pod \"kserve-controller-manager-7dcf4bff6-x262t\" (UID: \"e3a07917-7f99-4096-9836-ab43d55b4db8\") " pod="kserve/kserve-controller-manager-7dcf4bff6-x262t"
Sep 18 20:26:15.911229 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.911167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrm9c\" (UniqueName: \"kubernetes.io/projected/e3a07917-7f99-4096-9836-ab43d55b4db8-kube-api-access-xrm9c\") pod \"kserve-controller-manager-7dcf4bff6-x262t\" (UID: \"e3a07917-7f99-4096-9836-ab43d55b4db8\") " pod="kserve/kserve-controller-manager-7dcf4bff6-x262t"
Sep 18 20:26:15.911545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.911522    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3ef17b10-51f0-4100-9820-55c34ad598a5-data\") pod \"seaweedfs-64568bcd49-ts8m6\" (UID: \"3ef17b10-51f0-4100-9820-55c34ad598a5\") " pod="kserve/seaweedfs-64568bcd49-ts8m6"
Sep 18 20:26:15.913637 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.913619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd491e0b-5e5b-4c6d-a042-3d11f1fddf31-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-bhqdm\" (UID: \"dd491e0b-5e5b-4c6d-a042-3d11f1fddf31\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"
Sep 18 20:26:15.913804 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.913782    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3a07917-7f99-4096-9836-ab43d55b4db8-cert\") pod \"kserve-controller-manager-7dcf4bff6-x262t\" (UID: \"e3a07917-7f99-4096-9836-ab43d55b4db8\") " pod="kserve/kserve-controller-manager-7dcf4bff6-x262t"
Sep 18 20:26:15.919935 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.919876    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2m4z\" (UniqueName: \"kubernetes.io/projected/dd491e0b-5e5b-4c6d-a042-3d11f1fddf31-kube-api-access-v2m4z\") pod \"llmisvc-controller-manager-645f9c8fd6-bhqdm\" (UID: \"dd491e0b-5e5b-4c6d-a042-3d11f1fddf31\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"
Sep 18 20:26:15.920084 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.919885    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrm9c\" (UniqueName: \"kubernetes.io/projected/e3a07917-7f99-4096-9836-ab43d55b4db8-kube-api-access-xrm9c\") pod \"kserve-controller-manager-7dcf4bff6-x262t\" (UID: \"e3a07917-7f99-4096-9836-ab43d55b4db8\") " pod="kserve/kserve-controller-manager-7dcf4bff6-x262t"
Sep 18 20:26:15.920759 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.920738    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5b4k\" (UniqueName: \"kubernetes.io/projected/3ef17b10-51f0-4100-9820-55c34ad598a5-kube-api-access-h5b4k\") pod \"seaweedfs-64568bcd49-ts8m6\" (UID: \"3ef17b10-51f0-4100-9820-55c34ad598a5\") " pod="kserve/seaweedfs-64568bcd49-ts8m6"
Sep 18 20:26:15.980205 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.980182    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-x262t"
Sep 18 20:26:15.995295 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:15.995272    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"
Sep 18 20:26:16.021077 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:16.021016    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-ts8m6"
Sep 18 20:26:16.254481 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:16.254403    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-x262t"]
Sep 18 20:26:16.257409 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:26:16.257383    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a07917_7f99_4096_9836_ab43d55b4db8.slice/crio-b19d5abf3ee80b63ca8c86add0e4c42ac91ff8735fb1ecd7ea5bfe425d052c04 WatchSource:0}: Error finding container b19d5abf3ee80b63ca8c86add0e4c42ac91ff8735fb1ecd7ea5bfe425d052c04: Status 404 returned error can't find the container with id b19d5abf3ee80b63ca8c86add0e4c42ac91ff8735fb1ecd7ea5bfe425d052c04
Sep 18 20:26:16.258674 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:16.258654    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:26:16.275634 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:16.275613    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"]
Sep 18 20:26:16.277326 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:26:16.277303    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddd491e0b_5e5b_4c6d_a042_3d11f1fddf31.slice/crio-e5995f7aa6b35ebb8c6b071804a48bdcc3ec0c3c8b22d588d19c9234eefec914 WatchSource:0}: Error finding container e5995f7aa6b35ebb8c6b071804a48bdcc3ec0c3c8b22d588d19c9234eefec914: Status 404 returned error can't find the container with id e5995f7aa6b35ebb8c6b071804a48bdcc3ec0c3c8b22d588d19c9234eefec914
Sep 18 20:26:16.313156 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:16.313133    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-ts8m6"]
Sep 18 20:26:16.315487 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:26:16.315461    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef17b10_51f0_4100_9820_55c34ad598a5.slice/crio-3cfd105028d51b4beb97f18d324aef3918a653e496686b0220eb824a98e61299 WatchSource:0}: Error finding container 3cfd105028d51b4beb97f18d324aef3918a653e496686b0220eb824a98e61299: Status 404 returned error can't find the container with id 3cfd105028d51b4beb97f18d324aef3918a653e496686b0220eb824a98e61299
Sep 18 20:26:16.804327 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:16.804254    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-x262t" event={"ID":"e3a07917-7f99-4096-9836-ab43d55b4db8","Type":"ContainerStarted","Data":"b19d5abf3ee80b63ca8c86add0e4c42ac91ff8735fb1ecd7ea5bfe425d052c04"}
Sep 18 20:26:16.807502 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:16.807470    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-ts8m6" event={"ID":"3ef17b10-51f0-4100-9820-55c34ad598a5","Type":"ContainerStarted","Data":"3cfd105028d51b4beb97f18d324aef3918a653e496686b0220eb824a98e61299"}
Sep 18 20:26:16.809158 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:16.809079    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm" event={"ID":"dd491e0b-5e5b-4c6d-a042-3d11f1fddf31","Type":"ContainerStarted","Data":"e5995f7aa6b35ebb8c6b071804a48bdcc3ec0c3c8b22d588d19c9234eefec914"}
Sep 18 20:26:20.840841 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:20.840808    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-x262t" event={"ID":"e3a07917-7f99-4096-9836-ab43d55b4db8","Type":"ContainerStarted","Data":"579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7"}
Sep 18 20:26:20.841222 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:20.841069    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7dcf4bff6-x262t"
Sep 18 20:26:20.861101 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:20.861039    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7dcf4bff6-x262t" podStartSLOduration=1.435147595 podStartE2EDuration="5.86101577s" totalImagesPullingTime="4.425868175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:20.859847255 +0000 UTC m=+331.767749536" watchObservedRunningTime="2026-09-18 20:26:20.86101577 +0000 UTC m=+331.768918050"
Sep 18 20:26:21.851093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:21.851060    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-ts8m6" event={"ID":"3ef17b10-51f0-4100-9820-55c34ad598a5","Type":"ContainerStarted","Data":"3b84806aa55c81c6d6b59fe3c6c14263f6834535511f5cd9ce7a56c0420006f5"}
Sep 18 20:26:21.851570 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:21.851142    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-ts8m6"
Sep 18 20:26:21.852697 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:21.852671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm" event={"ID":"dd491e0b-5e5b-4c6d-a042-3d11f1fddf31","Type":"ContainerStarted","Data":"8affb51ad8749a508339756e0fa55b9eb4070bf15737e7f84bb85e2e24bbe2ec"}
Sep 18 20:26:21.852812 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:21.852756    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"
Sep 18 20:26:21.868731 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:21.868687    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-ts8m6" podStartSLOduration=2.401483458 podStartE2EDuration="6.868673508s" totalImagesPullingTime="4.46719005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:21.866716569 +0000 UTC m=+332.774618850" watchObservedRunningTime="2026-09-18 20:26:21.868673508 +0000 UTC m=+332.776575786"
Sep 18 20:26:21.883398 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:21.883358    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm" podStartSLOduration=2.542846547 podStartE2EDuration="6.883344185s" totalImagesPullingTime="4.340497638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:21.881718682 +0000 UTC m=+332.789620960" watchObservedRunningTime="2026-09-18 20:26:21.883344185 +0000 UTC m=+332.791246463"
Sep 18 20:26:27.860176 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:27.860117    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-ts8m6"
Sep 18 20:26:41.454412 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.454379    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d9ddcb57d-x8jtg"]
Sep 18 20:26:41.459911 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.459891    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.470661 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.470637    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9ddcb57d-x8jtg"]
Sep 18 20:26:41.524750 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.524723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-console-serving-cert\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.524885 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.524757    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-oauth-serving-cert\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.524885 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.524806    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-console-oauth-config\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.524885 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.524856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-console-config\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.524988 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.524884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-trusted-ca-bundle\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.524988 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.524902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjxd8\" (UniqueName: \"kubernetes.io/projected/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-kube-api-access-rjxd8\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.524988 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.524924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-service-ca\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.625447 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.625409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-console-config\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.625447 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.625450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-trusted-ca-bundle\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.625677 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.625474    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjxd8\" (UniqueName: \"kubernetes.io/projected/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-kube-api-access-rjxd8\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.625677 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.625503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-service-ca\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.625677 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.625535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-console-serving-cert\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.625677 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.625560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-oauth-serving-cert\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.625677 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.625626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-console-oauth-config\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.626184 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.626157    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-console-config\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.626358 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.626332    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-trusted-ca-bundle\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.626425 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.626383    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-service-ca\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.626425 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.626384    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-oauth-serving-cert\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.628105 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.628087    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-console-oauth-config\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.628194 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.628175    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-console-serving-cert\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.634371 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.634329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjxd8\" (UniqueName: \"kubernetes.io/projected/241d9e9c-0fc0-4189-a6b7-84cf0be0a364-kube-api-access-rjxd8\") pod \"console-5d9ddcb57d-x8jtg\" (UID: \"241d9e9c-0fc0-4189-a6b7-84cf0be0a364\") " pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:41.768931 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:41.768853    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:42.043511 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:42.043448    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9ddcb57d-x8jtg"]
Sep 18 20:26:42.045747 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:26:42.045719    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241d9e9c_0fc0_4189_a6b7_84cf0be0a364.slice/crio-deb629bb970901f3d97bcf6108caeec63a0bdea6034a776d5ed3f1acb31c2925 WatchSource:0}: Error finding container deb629bb970901f3d97bcf6108caeec63a0bdea6034a776d5ed3f1acb31c2925: Status 404 returned error can't find the container with id deb629bb970901f3d97bcf6108caeec63a0bdea6034a776d5ed3f1acb31c2925
Sep 18 20:26:42.949569 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:42.949532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9ddcb57d-x8jtg" event={"ID":"241d9e9c-0fc0-4189-a6b7-84cf0be0a364","Type":"ContainerStarted","Data":"044fef63d8624877c8e7758a23769445a80cd3e363743626042d1831115a3636"}
Sep 18 20:26:42.949981 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:42.949616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9ddcb57d-x8jtg" event={"ID":"241d9e9c-0fc0-4189-a6b7-84cf0be0a364","Type":"ContainerStarted","Data":"deb629bb970901f3d97bcf6108caeec63a0bdea6034a776d5ed3f1acb31c2925"}
Sep 18 20:26:42.968832 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:42.968789    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d9ddcb57d-x8jtg" podStartSLOduration=1.9687740900000001 podStartE2EDuration="1.96877409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:42.967486225 +0000 UTC m=+353.875388504" watchObservedRunningTime="2026-09-18 20:26:42.96877409 +0000 UTC m=+353.876676369"
Sep 18 20:26:51.769989 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:51.769951    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:51.770458 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:51.770014    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:51.774896 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:51.774875    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:51.857655 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:51.857633    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7dcf4bff6-x262t"
Sep 18 20:26:51.995227 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:51.995203    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d9ddcb57d-x8jtg"
Sep 18 20:26:52.099687 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:52.099601    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f9795bbb-nczwr"]
Sep 18 20:26:52.860388 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:52.860356    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bhqdm"
Sep 18 20:26:59.761069 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:59.761035    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-x262t"]
Sep 18 20:26:59.761595 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:59.761232    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7dcf4bff6-x262t" podUID="e3a07917-7f99-4096-9836-ab43d55b4db8" containerName="manager" containerID="cri-o://579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7" gracePeriod=10
Sep 18 20:26:59.790673 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:59.787663    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-q8flf"]
Sep 18 20:26:59.791711 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:59.791249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf"
Sep 18 20:26:59.796875 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:59.796853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-q8flf"]
Sep 18 20:26:59.872305 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:59.872277    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5vs6\" (UniqueName: \"kubernetes.io/projected/5436145d-779b-42ce-a406-53842b043fbc-kube-api-access-w5vs6\") pod \"kserve-controller-manager-7dcf4bff6-q8flf\" (UID: \"5436145d-779b-42ce-a406-53842b043fbc\") " pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf"
Sep 18 20:26:59.872403 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:59.872335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5436145d-779b-42ce-a406-53842b043fbc-cert\") pod \"kserve-controller-manager-7dcf4bff6-q8flf\" (UID: \"5436145d-779b-42ce-a406-53842b043fbc\") " pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf"
Sep 18 20:26:59.972784 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:59.972749    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5vs6\" (UniqueName: \"kubernetes.io/projected/5436145d-779b-42ce-a406-53842b043fbc-kube-api-access-w5vs6\") pod \"kserve-controller-manager-7dcf4bff6-q8flf\" (UID: \"5436145d-779b-42ce-a406-53842b043fbc\") " pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf"
Sep 18 20:26:59.972950 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:59.972809    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5436145d-779b-42ce-a406-53842b043fbc-cert\") pod \"kserve-controller-manager-7dcf4bff6-q8flf\" (UID: \"5436145d-779b-42ce-a406-53842b043fbc\") " pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf"
Sep 18 20:26:59.975205 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:59.975183    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5436145d-779b-42ce-a406-53842b043fbc-cert\") pod \"kserve-controller-manager-7dcf4bff6-q8flf\" (UID: \"5436145d-779b-42ce-a406-53842b043fbc\") " pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf"
Sep 18 20:26:59.980899 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:26:59.980867    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5vs6\" (UniqueName: \"kubernetes.io/projected/5436145d-779b-42ce-a406-53842b043fbc-kube-api-access-w5vs6\") pod \"kserve-controller-manager-7dcf4bff6-q8flf\" (UID: \"5436145d-779b-42ce-a406-53842b043fbc\") " pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf"
Sep 18 20:27:00.014412 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.014331    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-x262t"
Sep 18 20:27:00.028280 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.028253    2583 generic.go:356] "Generic (PLEG): container finished" podID="e3a07917-7f99-4096-9836-ab43d55b4db8" containerID="579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7" exitCode=0
Sep 18 20:27:00.028379 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.028291    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-x262t"
Sep 18 20:27:00.028379 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.028331    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-x262t" event={"ID":"e3a07917-7f99-4096-9836-ab43d55b4db8","Type":"ContainerDied","Data":"579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7"}
Sep 18 20:27:00.028451 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.028381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-x262t" event={"ID":"e3a07917-7f99-4096-9836-ab43d55b4db8","Type":"ContainerDied","Data":"b19d5abf3ee80b63ca8c86add0e4c42ac91ff8735fb1ecd7ea5bfe425d052c04"}
Sep 18 20:27:00.028451 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.028403    2583 scope.go:117] "RemoveContainer" containerID="579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7"
Sep 18 20:27:00.037624 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.037567    2583 scope.go:117] "RemoveContainer" containerID="579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7"
Sep 18 20:27:00.037912 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:27:00.037887    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7\": container with ID starting with 579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7 not found: ID does not exist" containerID="579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7"
Sep 18 20:27:00.038017 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.037923    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7"} err="failed to get container status \"579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7\": rpc error: code = NotFound desc = could not find container \"579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7\": container with ID starting with 579678d244513ae44daa22f3c384af83a530150fe406391a47388b312015e4c7 not found: ID does not exist"
Sep 18 20:27:00.073393 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.073363    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrm9c\" (UniqueName: \"kubernetes.io/projected/e3a07917-7f99-4096-9836-ab43d55b4db8-kube-api-access-xrm9c\") pod \"e3a07917-7f99-4096-9836-ab43d55b4db8\" (UID: \"e3a07917-7f99-4096-9836-ab43d55b4db8\") "
Sep 18 20:27:00.073553 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.073413    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3a07917-7f99-4096-9836-ab43d55b4db8-cert\") pod \"e3a07917-7f99-4096-9836-ab43d55b4db8\" (UID: \"e3a07917-7f99-4096-9836-ab43d55b4db8\") "
Sep 18 20:27:00.075614 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.075588    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a07917-7f99-4096-9836-ab43d55b4db8-cert" (OuterVolumeSpecName: "cert") pod "e3a07917-7f99-4096-9836-ab43d55b4db8" (UID: "e3a07917-7f99-4096-9836-ab43d55b4db8"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:27:00.075713 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.075655    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a07917-7f99-4096-9836-ab43d55b4db8-kube-api-access-xrm9c" (OuterVolumeSpecName: "kube-api-access-xrm9c") pod "e3a07917-7f99-4096-9836-ab43d55b4db8" (UID: "e3a07917-7f99-4096-9836-ab43d55b4db8"). InnerVolumeSpecName "kube-api-access-xrm9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:27:00.129311 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.129281    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf"
Sep 18 20:27:00.174741 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.174710    2583 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3a07917-7f99-4096-9836-ab43d55b4db8-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:27:00.174854 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.174750    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrm9c\" (UniqueName: \"kubernetes.io/projected/e3a07917-7f99-4096-9836-ab43d55b4db8-kube-api-access-xrm9c\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:27:00.351441 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.351414    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-x262t"]
Sep 18 20:27:00.355312 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.355285    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-x262t"]
Sep 18 20:27:00.397887 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:00.397864    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-q8flf"]
Sep 18 20:27:00.400270 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:27:00.400247    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5436145d_779b_42ce_a406_53842b043fbc.slice/crio-fc691cc7362820f20ec5d2022393e64a2415aefc2bb89899ce573e264ed34add WatchSource:0}: Error finding container fc691cc7362820f20ec5d2022393e64a2415aefc2bb89899ce573e264ed34add: Status 404 returned error can't find the container with id fc691cc7362820f20ec5d2022393e64a2415aefc2bb89899ce573e264ed34add
Sep 18 20:27:01.034519 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:01.034427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf" event={"ID":"5436145d-779b-42ce-a406-53842b043fbc","Type":"ContainerStarted","Data":"e3a19676b1c5ae66d418b6cdad47f95be08c7f5e281f692d27c16dd17e4a7eb7"}
Sep 18 20:27:01.034519 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:01.034489    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf" event={"ID":"5436145d-779b-42ce-a406-53842b043fbc","Type":"ContainerStarted","Data":"fc691cc7362820f20ec5d2022393e64a2415aefc2bb89899ce573e264ed34add"}
Sep 18 20:27:01.034988 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:01.034629    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf"
Sep 18 20:27:01.052680 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:01.052635    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf" podStartSLOduration=1.6546440009999999 podStartE2EDuration="2.052611722s" totalImagesPullingTime="397.967721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:27:01.051398062 +0000 UTC m=+371.959300341" watchObservedRunningTime="2026-09-18 20:27:01.052611722 +0000 UTC m=+371.960514033"
Sep 18 20:27:01.316147 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:01.316064    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3a07917-7f99-4096-9836-ab43d55b4db8" path="/var/lib/kubelet/pods/e3a07917-7f99-4096-9836-ab43d55b4db8/volumes"
Sep 18 20:27:17.119109 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.119074    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56f9795bbb-nczwr" podUID="f77e2af6-50bf-484f-97fa-6dd2fdfc04db" containerName="console" containerID="cri-o://7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1" gracePeriod=15
Sep 18 20:27:17.361488 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.361466    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:27:17.429341 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.429312    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-oauth-serving-cert\") pod \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") "
Sep 18 20:27:17.429483 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.429361    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-config\") pod \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") "
Sep 18 20:27:17.429552 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.429534    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-trusted-ca-bundle\") pod \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") "
Sep 18 20:27:17.429591 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.429576    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-serving-cert\") pod \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") "
Sep 18 20:27:17.429714 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.429681    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-config" (OuterVolumeSpecName: "console-config") pod "f77e2af6-50bf-484f-97fa-6dd2fdfc04db" (UID: "f77e2af6-50bf-484f-97fa-6dd2fdfc04db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:27:17.429714 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.429693    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f77e2af6-50bf-484f-97fa-6dd2fdfc04db" (UID: "f77e2af6-50bf-484f-97fa-6dd2fdfc04db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:27:17.429714 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.429709    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-service-ca\") pod \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") "
Sep 18 20:27:17.429915 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.429744    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94nc6\" (UniqueName: \"kubernetes.io/projected/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-kube-api-access-94nc6\") pod \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") "
Sep 18 20:27:17.429915 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.429787    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-oauth-config\") pod \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\" (UID: \"f77e2af6-50bf-484f-97fa-6dd2fdfc04db\") "
Sep 18 20:27:17.429915 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.429878    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f77e2af6-50bf-484f-97fa-6dd2fdfc04db" (UID: "f77e2af6-50bf-484f-97fa-6dd2fdfc04db"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:27:17.430065 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.430045    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-oauth-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:27:17.430065 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.430046    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-service-ca" (OuterVolumeSpecName: "service-ca") pod "f77e2af6-50bf-484f-97fa-6dd2fdfc04db" (UID: "f77e2af6-50bf-484f-97fa-6dd2fdfc04db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:27:17.430065 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.430059    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:27:17.430229 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.430068    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-trusted-ca-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:27:17.431796 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.431775    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f77e2af6-50bf-484f-97fa-6dd2fdfc04db" (UID: "f77e2af6-50bf-484f-97fa-6dd2fdfc04db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:27:17.431995 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.431971    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-kube-api-access-94nc6" (OuterVolumeSpecName: "kube-api-access-94nc6") pod "f77e2af6-50bf-484f-97fa-6dd2fdfc04db" (UID: "f77e2af6-50bf-484f-97fa-6dd2fdfc04db"). InnerVolumeSpecName "kube-api-access-94nc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:27:17.431995 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.431984    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f77e2af6-50bf-484f-97fa-6dd2fdfc04db" (UID: "f77e2af6-50bf-484f-97fa-6dd2fdfc04db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:27:17.530721 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.530667    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:27:17.530721 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.530716    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-service-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:27:17.530721 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.530725    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94nc6\" (UniqueName: \"kubernetes.io/projected/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-kube-api-access-94nc6\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:27:17.530721 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:17.530734    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f77e2af6-50bf-484f-97fa-6dd2fdfc04db-console-oauth-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:27:18.111079 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:18.111045    2583 generic.go:356] "Generic (PLEG): container finished" podID="f77e2af6-50bf-484f-97fa-6dd2fdfc04db" containerID="7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1" exitCode=0
Sep 18 20:27:18.111267 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:18.111185    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f9795bbb-nczwr"
Sep 18 20:27:18.111336 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:18.111183    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f9795bbb-nczwr" event={"ID":"f77e2af6-50bf-484f-97fa-6dd2fdfc04db","Type":"ContainerDied","Data":"7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1"}
Sep 18 20:27:18.111402 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:18.111390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f9795bbb-nczwr" event={"ID":"f77e2af6-50bf-484f-97fa-6dd2fdfc04db","Type":"ContainerDied","Data":"d548c91f073c2d2032dd7e9be693b4298f2943f711363d30135604503fbf0d20"}
Sep 18 20:27:18.111439 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:18.111416    2583 scope.go:117] "RemoveContainer" containerID="7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1"
Sep 18 20:27:18.121571 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:18.121203    2583 scope.go:117] "RemoveContainer" containerID="7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1"
Sep 18 20:27:18.121571 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:27:18.121509    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1\": container with ID starting with 7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1 not found: ID does not exist" containerID="7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1"
Sep 18 20:27:18.121571 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:18.121542    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1"} err="failed to get container status \"7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1\": rpc error: code = NotFound desc = could not find container \"7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1\": container with ID starting with 7cfdd74eb899b70ba9acf09a34287d71b6373a83bb741f8ce5ca200631ff3dc1 not found: ID does not exist"
Sep 18 20:27:18.137765 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:18.137736    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f9795bbb-nczwr"]
Sep 18 20:27:18.141520 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:18.141498    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56f9795bbb-nczwr"]
Sep 18 20:27:19.316965 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:19.316929    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f77e2af6-50bf-484f-97fa-6dd2fdfc04db" path="/var/lib/kubelet/pods/f77e2af6-50bf-484f-97fa-6dd2fdfc04db/volumes"
Sep 18 20:27:32.044380 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:27:32.044350    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7dcf4bff6-q8flf"
Sep 18 20:29:30.202787 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.202711    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-knh92"]
Sep 18 20:29:30.203254 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.203042    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f77e2af6-50bf-484f-97fa-6dd2fdfc04db" containerName="console"
Sep 18 20:29:30.203254 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.203052    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f77e2af6-50bf-484f-97fa-6dd2fdfc04db" containerName="console"
Sep 18 20:29:30.203254 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.203079    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a07917-7f99-4096-9836-ab43d55b4db8" containerName="manager"
Sep 18 20:29:30.203254 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.203083    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a07917-7f99-4096-9836-ab43d55b4db8" containerName="manager"
Sep 18 20:29:30.203254 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.203154    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f77e2af6-50bf-484f-97fa-6dd2fdfc04db" containerName="console"
Sep 18 20:29:30.203254 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.203164    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a07917-7f99-4096-9836-ab43d55b4db8" containerName="manager"
Sep 18 20:29:30.205027 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.205010    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92"
Sep 18 20:29:30.207856 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.207831    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 18 20:29:30.207980 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.207858    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 18 20:29:30.216366 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.216341    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-knh92"]
Sep 18 20:29:30.232667 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.232630    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-298sv\" (UniqueName: \"kubernetes.io/projected/4be1e359-c781-44e2-afc1-b054332f1663-kube-api-access-298sv\") pod \"seaweedfs-tls-serving-67b7477c6d-knh92\" (UID: \"4be1e359-c781-44e2-afc1-b054332f1663\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92"
Sep 18 20:29:30.232771 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.232690    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/4be1e359-c781-44e2-afc1-b054332f1663-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-knh92\" (UID: \"4be1e359-c781-44e2-afc1-b054332f1663\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92"
Sep 18 20:29:30.232771 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.232719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4be1e359-c781-44e2-afc1-b054332f1663-data\") pod \"seaweedfs-tls-serving-67b7477c6d-knh92\" (UID: \"4be1e359-c781-44e2-afc1-b054332f1663\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92"
Sep 18 20:29:30.333809 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.333781    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298sv\" (UniqueName: \"kubernetes.io/projected/4be1e359-c781-44e2-afc1-b054332f1663-kube-api-access-298sv\") pod \"seaweedfs-tls-serving-67b7477c6d-knh92\" (UID: \"4be1e359-c781-44e2-afc1-b054332f1663\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92"
Sep 18 20:29:30.333963 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.333836    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/4be1e359-c781-44e2-afc1-b054332f1663-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-knh92\" (UID: \"4be1e359-c781-44e2-afc1-b054332f1663\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92"
Sep 18 20:29:30.333963 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.333870    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4be1e359-c781-44e2-afc1-b054332f1663-data\") pod \"seaweedfs-tls-serving-67b7477c6d-knh92\" (UID: \"4be1e359-c781-44e2-afc1-b054332f1663\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92"
Sep 18 20:29:30.334257 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.334241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4be1e359-c781-44e2-afc1-b054332f1663-data\") pod \"seaweedfs-tls-serving-67b7477c6d-knh92\" (UID: \"4be1e359-c781-44e2-afc1-b054332f1663\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92"
Sep 18 20:29:30.336480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.336460    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/4be1e359-c781-44e2-afc1-b054332f1663-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-knh92\" (UID: \"4be1e359-c781-44e2-afc1-b054332f1663\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92"
Sep 18 20:29:30.342305 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.342280    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-298sv\" (UniqueName: \"kubernetes.io/projected/4be1e359-c781-44e2-afc1-b054332f1663-kube-api-access-298sv\") pod \"seaweedfs-tls-serving-67b7477c6d-knh92\" (UID: \"4be1e359-c781-44e2-afc1-b054332f1663\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92"
Sep 18 20:29:30.513649 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.513567    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92"
Sep 18 20:29:30.784382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:30.784352    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-knh92"]
Sep 18 20:29:30.786890 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:29:30.786859    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be1e359_c781_44e2_afc1_b054332f1663.slice/crio-6e125031c5f65d6799cea8b1299be37d394c3c3180813ba26248e7eeaff2f4e7 WatchSource:0}: Error finding container 6e125031c5f65d6799cea8b1299be37d394c3c3180813ba26248e7eeaff2f4e7: Status 404 returned error can't find the container with id 6e125031c5f65d6799cea8b1299be37d394c3c3180813ba26248e7eeaff2f4e7
Sep 18 20:29:31.707156 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:31.707111    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92" event={"ID":"4be1e359-c781-44e2-afc1-b054332f1663","Type":"ContainerStarted","Data":"badf7bdef5cbad17a3137d7db172a5f3b93f1e5897ac780861d589d8ab801514"}
Sep 18 20:29:31.707156 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:31.707170    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92" event={"ID":"4be1e359-c781-44e2-afc1-b054332f1663","Type":"ContainerStarted","Data":"6e125031c5f65d6799cea8b1299be37d394c3c3180813ba26248e7eeaff2f4e7"}
Sep 18 20:29:31.723356 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:29:31.723291    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-knh92" podStartSLOduration=1.478071103 podStartE2EDuration="1.723269311s" totalImagesPullingTime="245.198208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:31.722938766 +0000 UTC m=+522.630841046" watchObservedRunningTime="2026-09-18 20:29:31.723269311 +0000 UTC m=+522.631171591"
Sep 18 20:30:11.315929 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.315897    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"]
Sep 18 20:30:11.319484 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.319464    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.322672 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.322648    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:30:11.323312 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.323291    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 18 20:30:11.323458 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.323368    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:30:11.323458 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.323279    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 18 20:30:11.323576 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.323491    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:30:11.323780 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.323761    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:30:11.323905 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.323761    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:30:11.326137 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.326101    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"]
Sep 18 20:30:11.471788 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.471754    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppg9x\" (UniqueName: \"kubernetes.io/projected/34311bc4-e51e-41e4-bd4e-c52f0e691626-kube-api-access-ppg9x\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.471968 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.471792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34311bc4-e51e-41e4-bd4e-c52f0e691626-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.471968 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.471841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34311bc4-e51e-41e4-bd4e-c52f0e691626-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.471968 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.471928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34311bc4-e51e-41e4-bd4e-c52f0e691626-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.471968 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.471956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34311bc4-e51e-41e4-bd4e-c52f0e691626-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.572979 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.572894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34311bc4-e51e-41e4-bd4e-c52f0e691626-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.572979 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.572929    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34311bc4-e51e-41e4-bd4e-c52f0e691626-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.572979 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.572971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppg9x\" (UniqueName: \"kubernetes.io/projected/34311bc4-e51e-41e4-bd4e-c52f0e691626-kube-api-access-ppg9x\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.573271 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.572990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34311bc4-e51e-41e4-bd4e-c52f0e691626-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.573271 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.573030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34311bc4-e51e-41e4-bd4e-c52f0e691626-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.573560 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.573541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34311bc4-e51e-41e4-bd4e-c52f0e691626-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.573680 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.573661    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34311bc4-e51e-41e4-bd4e-c52f0e691626-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.573743 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.573724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34311bc4-e51e-41e4-bd4e-c52f0e691626-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.575822 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.575799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34311bc4-e51e-41e4-bd4e-c52f0e691626-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.580786 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.580768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppg9x\" (UniqueName: \"kubernetes.io/projected/34311bc4-e51e-41e4-bd4e-c52f0e691626-kube-api-access-ppg9x\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-vzbrv\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.631635 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.631608    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:11.902971 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:11.902947    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"]
Sep 18 20:30:11.905489 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:30:11.905463    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34311bc4_e51e_41e4_bd4e_c52f0e691626.slice/crio-32503b53f3a74304fd53027e3d79e500431213aec56d11dc4c92e79e21856e83 WatchSource:0}: Error finding container 32503b53f3a74304fd53027e3d79e500431213aec56d11dc4c92e79e21856e83: Status 404 returned error can't find the container with id 32503b53f3a74304fd53027e3d79e500431213aec56d11dc4c92e79e21856e83
Sep 18 20:30:12.895416 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:12.895384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" event={"ID":"34311bc4-e51e-41e4-bd4e-c52f0e691626","Type":"ContainerStarted","Data":"32503b53f3a74304fd53027e3d79e500431213aec56d11dc4c92e79e21856e83"}
Sep 18 20:30:15.911577 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:15.911538    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" event={"ID":"34311bc4-e51e-41e4-bd4e-c52f0e691626","Type":"ContainerStarted","Data":"2ceec8c3885b7abfc37ba4765b4186ad76bee380ccf97e8d07873b3de53afc6a"}
Sep 18 20:30:19.929464 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:19.929430    2583 generic.go:356] "Generic (PLEG): container finished" podID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerID="2ceec8c3885b7abfc37ba4765b4186ad76bee380ccf97e8d07873b3de53afc6a" exitCode=0
Sep 18 20:30:19.929867 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:19.929508    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" event={"ID":"34311bc4-e51e-41e4-bd4e-c52f0e691626","Type":"ContainerDied","Data":"2ceec8c3885b7abfc37ba4765b4186ad76bee380ccf97e8d07873b3de53afc6a"}
Sep 18 20:30:43.066156 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:43.066104    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" event={"ID":"34311bc4-e51e-41e4-bd4e-c52f0e691626","Type":"ContainerStarted","Data":"52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7"}
Sep 18 20:30:47.087281 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:47.087243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" event={"ID":"34311bc4-e51e-41e4-bd4e-c52f0e691626","Type":"ContainerStarted","Data":"cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac"}
Sep 18 20:30:47.087635 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:47.087370    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:47.108582 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:47.108532    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podStartSLOduration=2.728270699 podStartE2EDuration="36.108517892s" totalImagesPullingTime="29.542695032s" totalInitContainerRuntime="3.837552161s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:30:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:30:47.10585317 +0000 UTC m=+598.013755450" watchObservedRunningTime="2026-09-18 20:30:47.108517892 +0000 UTC m=+598.016420170"
Sep 18 20:30:48.098370 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:48.098335    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:48.099891 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:48.099857    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:30:49.103000 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:49.102961    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:30:51.241605 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:51.241573    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:30:51.242300 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:51.242278    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:30:54.108256 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:54.108228    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:30:54.108780 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:30:54.108752    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:31:04.109111 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:31:04.109028    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:31:14.108885 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:31:14.108845    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:31:24.109487 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:31:24.109446    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:31:34.109563 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:31:34.109523    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:31:44.109250 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:31:44.109213    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:31:54.109567 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:31:54.109520    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:32:04.110110 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:04.110076    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:32:11.848884 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:11.848842    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"]
Sep 18 20:32:11.849384 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:11.849079    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container" containerID="cri-o://52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7" gracePeriod=30
Sep 18 20:32:11.849384 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:11.849103    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kube-rbac-proxy" containerID="cri-o://cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac" gracePeriod=30
Sep 18 20:32:12.487043 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:12.487012    2583 generic.go:356] "Generic (PLEG): container finished" podID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerID="cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac" exitCode=2
Sep 18 20:32:12.487233 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:12.487065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" event={"ID":"34311bc4-e51e-41e4-bd4e-c52f0e691626","Type":"ContainerDied","Data":"cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac"}
Sep 18 20:32:14.103344 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:14.103298    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 18 20:32:14.108980 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:14.108953    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:32:16.791104 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.791069    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:32:16.881384 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.881308    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34311bc4-e51e-41e4-bd4e-c52f0e691626-proxy-tls\") pod \"34311bc4-e51e-41e4-bd4e-c52f0e691626\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") "
Sep 18 20:32:16.881384 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.881345    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34311bc4-e51e-41e4-bd4e-c52f0e691626-kube-rbac-proxy-sar-config\") pod \"34311bc4-e51e-41e4-bd4e-c52f0e691626\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") "
Sep 18 20:32:16.881384 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.881380    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34311bc4-e51e-41e4-bd4e-c52f0e691626-kserve-provision-location\") pod \"34311bc4-e51e-41e4-bd4e-c52f0e691626\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") "
Sep 18 20:32:16.881683 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.881405    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34311bc4-e51e-41e4-bd4e-c52f0e691626-cabundle-cert\") pod \"34311bc4-e51e-41e4-bd4e-c52f0e691626\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") "
Sep 18 20:32:16.881683 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.881482    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppg9x\" (UniqueName: \"kubernetes.io/projected/34311bc4-e51e-41e4-bd4e-c52f0e691626-kube-api-access-ppg9x\") pod \"34311bc4-e51e-41e4-bd4e-c52f0e691626\" (UID: \"34311bc4-e51e-41e4-bd4e-c52f0e691626\") "
Sep 18 20:32:16.881789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.881722    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34311bc4-e51e-41e4-bd4e-c52f0e691626-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34311bc4-e51e-41e4-bd4e-c52f0e691626" (UID: "34311bc4-e51e-41e4-bd4e-c52f0e691626"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:16.881863 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.881838    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34311bc4-e51e-41e4-bd4e-c52f0e691626-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34311bc4-e51e-41e4-bd4e-c52f0e691626" (UID: "34311bc4-e51e-41e4-bd4e-c52f0e691626"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:16.881942 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.881917    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34311bc4-e51e-41e4-bd4e-c52f0e691626-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34311bc4-e51e-41e4-bd4e-c52f0e691626" (UID: "34311bc4-e51e-41e4-bd4e-c52f0e691626"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:16.883821 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.883797    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34311bc4-e51e-41e4-bd4e-c52f0e691626-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34311bc4-e51e-41e4-bd4e-c52f0e691626" (UID: "34311bc4-e51e-41e4-bd4e-c52f0e691626"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:32:16.883821 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.883810    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34311bc4-e51e-41e4-bd4e-c52f0e691626-kube-api-access-ppg9x" (OuterVolumeSpecName: "kube-api-access-ppg9x") pod "34311bc4-e51e-41e4-bd4e-c52f0e691626" (UID: "34311bc4-e51e-41e4-bd4e-c52f0e691626"). InnerVolumeSpecName "kube-api-access-ppg9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:16.982361 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.982327    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34311bc4-e51e-41e4-bd4e-c52f0e691626-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:32:16.982361 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.982360    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppg9x\" (UniqueName: \"kubernetes.io/projected/34311bc4-e51e-41e4-bd4e-c52f0e691626-kube-api-access-ppg9x\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:32:16.982578 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.982372    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34311bc4-e51e-41e4-bd4e-c52f0e691626-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:32:16.982578 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.982382    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34311bc4-e51e-41e4-bd4e-c52f0e691626-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:32:16.982578 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:16.982390    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34311bc4-e51e-41e4-bd4e-c52f0e691626-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:32:17.049366 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.049331    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"]
Sep 18 20:32:17.049777 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.049759    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kube-rbac-proxy"
Sep 18 20:32:17.049865 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.049781    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kube-rbac-proxy"
Sep 18 20:32:17.049865 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.049795    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="storage-initializer"
Sep 18 20:32:17.049865 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.049802    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="storage-initializer"
Sep 18 20:32:17.049865 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.049811    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container"
Sep 18 20:32:17.049865 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.049821    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container"
Sep 18 20:32:17.050110 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.049907    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kube-rbac-proxy"
Sep 18 20:32:17.050110 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.049922    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerName="kserve-container"
Sep 18 20:32:17.052384 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.052360    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.055116 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.055096    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 18 20:32:17.055333 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.055310    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 18 20:32:17.064764 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.064738    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"]
Sep 18 20:32:17.184031 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.184002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e2ca9e-2498-4994-9742-ec3d2e520548-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.184235 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.184056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f57f5\" (UniqueName: \"kubernetes.io/projected/44e2ca9e-2498-4994-9742-ec3d2e520548-kube-api-access-f57f5\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.184235 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.184080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44e2ca9e-2498-4994-9742-ec3d2e520548-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.184235 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.184173    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44e2ca9e-2498-4994-9742-ec3d2e520548-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.184383 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.184254    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44e2ca9e-2498-4994-9742-ec3d2e520548-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.285013 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.284980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44e2ca9e-2498-4994-9742-ec3d2e520548-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.285238 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.285031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44e2ca9e-2498-4994-9742-ec3d2e520548-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.285238 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.285171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e2ca9e-2498-4994-9742-ec3d2e520548-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.285370 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.285252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f57f5\" (UniqueName: \"kubernetes.io/projected/44e2ca9e-2498-4994-9742-ec3d2e520548-kube-api-access-f57f5\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.285370 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.285284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44e2ca9e-2498-4994-9742-ec3d2e520548-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.285764 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.285732    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44e2ca9e-2498-4994-9742-ec3d2e520548-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.285918 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.285895    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44e2ca9e-2498-4994-9742-ec3d2e520548-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.285989 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.285907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44e2ca9e-2498-4994-9742-ec3d2e520548-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.287790 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.287771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e2ca9e-2498-4994-9742-ec3d2e520548-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.293568 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.293542    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f57f5\" (UniqueName: \"kubernetes.io/projected/44e2ca9e-2498-4994-9742-ec3d2e520548-kube-api-access-f57f5\") pod \"isvc-predictive-xgboost-predictor-695f978978-nfwn2\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.363747 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.363722    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:17.514696 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.514597    2583 generic.go:356] "Generic (PLEG): container finished" podID="34311bc4-e51e-41e4-bd4e-c52f0e691626" containerID="52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7" exitCode=0
Sep 18 20:32:17.514696 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.514678    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"
Sep 18 20:32:17.514917 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.514694    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" event={"ID":"34311bc4-e51e-41e4-bd4e-c52f0e691626","Type":"ContainerDied","Data":"52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7"}
Sep 18 20:32:17.514917 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.514799    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv" event={"ID":"34311bc4-e51e-41e4-bd4e-c52f0e691626","Type":"ContainerDied","Data":"32503b53f3a74304fd53027e3d79e500431213aec56d11dc4c92e79e21856e83"}
Sep 18 20:32:17.514917 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.514840    2583 scope.go:117] "RemoveContainer" containerID="cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac"
Sep 18 20:32:17.523456 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.523440    2583 scope.go:117] "RemoveContainer" containerID="52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7"
Sep 18 20:32:17.530979 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.530960    2583 scope.go:117] "RemoveContainer" containerID="2ceec8c3885b7abfc37ba4765b4186ad76bee380ccf97e8d07873b3de53afc6a"
Sep 18 20:32:17.534656 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.534636    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"]
Sep 18 20:32:17.538518 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.538494    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-vzbrv"]
Sep 18 20:32:17.539107 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.539080    2583 scope.go:117] "RemoveContainer" containerID="cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac"
Sep 18 20:32:17.539399 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:32:17.539380    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac\": container with ID starting with cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac not found: ID does not exist" containerID="cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac"
Sep 18 20:32:17.539462 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.539407    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac"} err="failed to get container status \"cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac\": rpc error: code = NotFound desc = could not find container \"cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac\": container with ID starting with cbdd7fed676437d1f1e612df42f23f107f171ed603fcfc887450309509f0f2ac not found: ID does not exist"
Sep 18 20:32:17.539462 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.539424    2583 scope.go:117] "RemoveContainer" containerID="52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7"
Sep 18 20:32:17.539656 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:32:17.539641    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7\": container with ID starting with 52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7 not found: ID does not exist" containerID="52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7"
Sep 18 20:32:17.539704 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.539661    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7"} err="failed to get container status \"52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7\": rpc error: code = NotFound desc = could not find container \"52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7\": container with ID starting with 52682ed768feda7d3d7b4a86b8a4b52acccac210f31835bab3a3df5b6dc1ddc7 not found: ID does not exist"
Sep 18 20:32:17.539704 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.539673    2583 scope.go:117] "RemoveContainer" containerID="2ceec8c3885b7abfc37ba4765b4186ad76bee380ccf97e8d07873b3de53afc6a"
Sep 18 20:32:17.539913 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:32:17.539895    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ceec8c3885b7abfc37ba4765b4186ad76bee380ccf97e8d07873b3de53afc6a\": container with ID starting with 2ceec8c3885b7abfc37ba4765b4186ad76bee380ccf97e8d07873b3de53afc6a not found: ID does not exist" containerID="2ceec8c3885b7abfc37ba4765b4186ad76bee380ccf97e8d07873b3de53afc6a"
Sep 18 20:32:17.539960 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.539916    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ceec8c3885b7abfc37ba4765b4186ad76bee380ccf97e8d07873b3de53afc6a"} err="failed to get container status \"2ceec8c3885b7abfc37ba4765b4186ad76bee380ccf97e8d07873b3de53afc6a\": rpc error: code = NotFound desc = could not find container \"2ceec8c3885b7abfc37ba4765b4186ad76bee380ccf97e8d07873b3de53afc6a\": container with ID starting with 2ceec8c3885b7abfc37ba4765b4186ad76bee380ccf97e8d07873b3de53afc6a not found: ID does not exist"
Sep 18 20:32:17.635901 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:17.635876    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"]
Sep 18 20:32:17.638046 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:32:17.638017    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e2ca9e_2498_4994_9742_ec3d2e520548.slice/crio-7e6d36bbf41cabb320eff03c1aa6ba4b62759aab547d47560daf69c3a619539d WatchSource:0}: Error finding container 7e6d36bbf41cabb320eff03c1aa6ba4b62759aab547d47560daf69c3a619539d: Status 404 returned error can't find the container with id 7e6d36bbf41cabb320eff03c1aa6ba4b62759aab547d47560daf69c3a619539d
Sep 18 20:32:18.520939 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:18.520905    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" event={"ID":"44e2ca9e-2498-4994-9742-ec3d2e520548","Type":"ContainerStarted","Data":"531fbf735c2138cf623a22e75f3000a9eb8652f25f52d815526ced3a668ad71b"}
Sep 18 20:32:18.521405 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:18.520991    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" event={"ID":"44e2ca9e-2498-4994-9742-ec3d2e520548","Type":"ContainerStarted","Data":"7e6d36bbf41cabb320eff03c1aa6ba4b62759aab547d47560daf69c3a619539d"}
Sep 18 20:32:19.316778 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:19.316748    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34311bc4-e51e-41e4-bd4e-c52f0e691626" path="/var/lib/kubelet/pods/34311bc4-e51e-41e4-bd4e-c52f0e691626/volumes"
Sep 18 20:32:22.542317 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:22.542290    2583 generic.go:356] "Generic (PLEG): container finished" podID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerID="531fbf735c2138cf623a22e75f3000a9eb8652f25f52d815526ced3a668ad71b" exitCode=0
Sep 18 20:32:22.542713 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:22.542362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" event={"ID":"44e2ca9e-2498-4994-9742-ec3d2e520548","Type":"ContainerDied","Data":"531fbf735c2138cf623a22e75f3000a9eb8652f25f52d815526ced3a668ad71b"}
Sep 18 20:32:23.549000 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:23.548966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" event={"ID":"44e2ca9e-2498-4994-9742-ec3d2e520548","Type":"ContainerStarted","Data":"8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5"}
Sep 18 20:32:23.549411 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:23.549028    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" event={"ID":"44e2ca9e-2498-4994-9742-ec3d2e520548","Type":"ContainerStarted","Data":"a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672"}
Sep 18 20:32:23.549411 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:23.549238    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:23.573657 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:23.573608    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podStartSLOduration=2.661049288 podStartE2EDuration="6.573593729s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.912544441s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:23.572007978 +0000 UTC m=+694.479910263" watchObservedRunningTime="2026-09-18 20:32:23.573593729 +0000 UTC m=+694.481496008"
Sep 18 20:32:24.557186 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:24.557158    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:24.558704 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:24.558666    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:32:25.561606 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:25.561565    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:32:30.566156 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:30.566060    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:32:30.566571 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:30.566522    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:32:40.566827 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:40.566784    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:32:50.566503 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:32:50.566460    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:33:00.566774 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:00.566736    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:33:10.567445 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:10.567405    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:33:20.567231 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:20.567189    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:33:30.567176 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:30.567107    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:33:39.316305 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:39.316279    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:33:47.545336 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:47.545305    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"]
Sep 18 20:33:47.545887 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:47.545521    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container" containerID="cri-o://a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672" gracePeriod=30
Sep 18 20:33:47.545887 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:47.545570    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kube-rbac-proxy" containerID="cri-o://8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5" gracePeriod=30
Sep 18 20:33:47.937514 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:47.937481    2583 generic.go:356] "Generic (PLEG): container finished" podID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerID="8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5" exitCode=2
Sep 18 20:33:47.937678 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:47.937553    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" event={"ID":"44e2ca9e-2498-4994-9742-ec3d2e520548","Type":"ContainerDied","Data":"8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5"}
Sep 18 20:33:49.310559 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:49.310521    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:33:50.561921 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:50.561873    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 18 20:33:52.490173 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.490149    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:33:52.624189 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.624087    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f57f5\" (UniqueName: \"kubernetes.io/projected/44e2ca9e-2498-4994-9742-ec3d2e520548-kube-api-access-f57f5\") pod \"44e2ca9e-2498-4994-9742-ec3d2e520548\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") "
Sep 18 20:33:52.624189 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.624181    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44e2ca9e-2498-4994-9742-ec3d2e520548-cabundle-cert\") pod \"44e2ca9e-2498-4994-9742-ec3d2e520548\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") "
Sep 18 20:33:52.624398 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.624212    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e2ca9e-2498-4994-9742-ec3d2e520548-proxy-tls\") pod \"44e2ca9e-2498-4994-9742-ec3d2e520548\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") "
Sep 18 20:33:52.624398 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.624247    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44e2ca9e-2498-4994-9742-ec3d2e520548-kserve-provision-location\") pod \"44e2ca9e-2498-4994-9742-ec3d2e520548\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") "
Sep 18 20:33:52.624398 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.624310    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44e2ca9e-2498-4994-9742-ec3d2e520548-kube-rbac-proxy-sar-config\") pod \"44e2ca9e-2498-4994-9742-ec3d2e520548\" (UID: \"44e2ca9e-2498-4994-9742-ec3d2e520548\") "
Sep 18 20:33:52.624567 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.624526    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e2ca9e-2498-4994-9742-ec3d2e520548-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "44e2ca9e-2498-4994-9742-ec3d2e520548" (UID: "44e2ca9e-2498-4994-9742-ec3d2e520548"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:33:52.624753 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.624719    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e2ca9e-2498-4994-9742-ec3d2e520548-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44e2ca9e-2498-4994-9742-ec3d2e520548" (UID: "44e2ca9e-2498-4994-9742-ec3d2e520548"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:33:52.624857 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.624814    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e2ca9e-2498-4994-9742-ec3d2e520548-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44e2ca9e-2498-4994-9742-ec3d2e520548" (UID: "44e2ca9e-2498-4994-9742-ec3d2e520548"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:33:52.626403 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.626381    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e2ca9e-2498-4994-9742-ec3d2e520548-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44e2ca9e-2498-4994-9742-ec3d2e520548" (UID: "44e2ca9e-2498-4994-9742-ec3d2e520548"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:33:52.626531 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.626512    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e2ca9e-2498-4994-9742-ec3d2e520548-kube-api-access-f57f5" (OuterVolumeSpecName: "kube-api-access-f57f5") pod "44e2ca9e-2498-4994-9742-ec3d2e520548" (UID: "44e2ca9e-2498-4994-9742-ec3d2e520548"). InnerVolumeSpecName "kube-api-access-f57f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:33:52.725191 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.725146    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f57f5\" (UniqueName: \"kubernetes.io/projected/44e2ca9e-2498-4994-9742-ec3d2e520548-kube-api-access-f57f5\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:33:52.725191 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.725183    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44e2ca9e-2498-4994-9742-ec3d2e520548-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:33:52.725191 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.725195    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e2ca9e-2498-4994-9742-ec3d2e520548-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:33:52.725408 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.725206    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44e2ca9e-2498-4994-9742-ec3d2e520548-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:33:52.725408 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.725218    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44e2ca9e-2498-4994-9742-ec3d2e520548-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:33:52.743262 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.743224    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"]
Sep 18 20:33:52.743641 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.743623    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="storage-initializer"
Sep 18 20:33:52.743720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.743644    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="storage-initializer"
Sep 18 20:33:52.743720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.743687    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container"
Sep 18 20:33:52.743720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.743696    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container"
Sep 18 20:33:52.743720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.743705    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kube-rbac-proxy"
Sep 18 20:33:52.743720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.743713    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kube-rbac-proxy"
Sep 18 20:33:52.743953 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.743798    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kube-rbac-proxy"
Sep 18 20:33:52.743953 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.743809    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerName="kserve-container"
Sep 18 20:33:52.747061 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.747041    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.749720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.749699    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 18 20:33:52.749796 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.749719    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 18 20:33:52.756789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.756764    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"]
Sep 18 20:33:52.826165 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.826112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.826299 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.826177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.826299 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.826206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.826299 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.826266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2wz\" (UniqueName: \"kubernetes.io/projected/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kube-api-access-pp2wz\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.826299 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.826296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.927174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.927145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.927379 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.927191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.927379 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.927211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.927379 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.927276    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2wz\" (UniqueName: \"kubernetes.io/projected/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kube-api-access-pp2wz\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.927379 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.927321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.927569 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.927549    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.927871 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.927851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.927914 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.927898    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.929759 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.929740    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.936095 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.936077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2wz\" (UniqueName: \"kubernetes.io/projected/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kube-api-access-pp2wz\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:52.961497 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.961476    2583 generic.go:356] "Generic (PLEG): container finished" podID="44e2ca9e-2498-4994-9742-ec3d2e520548" containerID="a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672" exitCode=0
Sep 18 20:33:52.961596 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.961505    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" event={"ID":"44e2ca9e-2498-4994-9742-ec3d2e520548","Type":"ContainerDied","Data":"a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672"}
Sep 18 20:33:52.961596 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.961528    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"
Sep 18 20:33:52.961596 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.961562    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2" event={"ID":"44e2ca9e-2498-4994-9742-ec3d2e520548","Type":"ContainerDied","Data":"7e6d36bbf41cabb320eff03c1aa6ba4b62759aab547d47560daf69c3a619539d"}
Sep 18 20:33:52.961596 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.961587    2583 scope.go:117] "RemoveContainer" containerID="8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5"
Sep 18 20:33:52.970326 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.970311    2583 scope.go:117] "RemoveContainer" containerID="a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672"
Sep 18 20:33:52.977844 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.977829    2583 scope.go:117] "RemoveContainer" containerID="531fbf735c2138cf623a22e75f3000a9eb8652f25f52d815526ced3a668ad71b"
Sep 18 20:33:52.985670 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.985647    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"]
Sep 18 20:33:52.987106 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.987087    2583 scope.go:117] "RemoveContainer" containerID="8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5"
Sep 18 20:33:52.987425 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:33:52.987405    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5\": container with ID starting with 8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5 not found: ID does not exist" containerID="8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5"
Sep 18 20:33:52.987530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.987435    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5"} err="failed to get container status \"8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5\": rpc error: code = NotFound desc = could not find container \"8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5\": container with ID starting with 8ee68df4fbaa1238fb11666a36dfbca03296a4e5fff29940c7578849b874f2a5 not found: ID does not exist"
Sep 18 20:33:52.987530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.987459    2583 scope.go:117] "RemoveContainer" containerID="a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672"
Sep 18 20:33:52.987789 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:33:52.987772    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672\": container with ID starting with a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672 not found: ID does not exist" containerID="a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672"
Sep 18 20:33:52.987851 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.987795    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672"} err="failed to get container status \"a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672\": rpc error: code = NotFound desc = could not find container \"a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672\": container with ID starting with a8c59ca9cec7942119a66866213a5c1df6223448d83c5b491d0f9bd941b74672 not found: ID does not exist"
Sep 18 20:33:52.987851 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.987811    2583 scope.go:117] "RemoveContainer" containerID="531fbf735c2138cf623a22e75f3000a9eb8652f25f52d815526ced3a668ad71b"
Sep 18 20:33:52.988074 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:33:52.988057    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531fbf735c2138cf623a22e75f3000a9eb8652f25f52d815526ced3a668ad71b\": container with ID starting with 531fbf735c2138cf623a22e75f3000a9eb8652f25f52d815526ced3a668ad71b not found: ID does not exist" containerID="531fbf735c2138cf623a22e75f3000a9eb8652f25f52d815526ced3a668ad71b"
Sep 18 20:33:52.988155 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.988082    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531fbf735c2138cf623a22e75f3000a9eb8652f25f52d815526ced3a668ad71b"} err="failed to get container status \"531fbf735c2138cf623a22e75f3000a9eb8652f25f52d815526ced3a668ad71b\": rpc error: code = NotFound desc = could not find container \"531fbf735c2138cf623a22e75f3000a9eb8652f25f52d815526ced3a668ad71b\": container with ID starting with 531fbf735c2138cf623a22e75f3000a9eb8652f25f52d815526ced3a668ad71b not found: ID does not exist"
Sep 18 20:33:52.988438 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:52.988422    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-nfwn2"]
Sep 18 20:33:53.058849 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:53.058815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:53.322452 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:53.322382    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44e2ca9e-2498-4994-9742-ec3d2e520548" path="/var/lib/kubelet/pods/44e2ca9e-2498-4994-9742-ec3d2e520548/volumes"
Sep 18 20:33:53.333581 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:53.333551    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"]
Sep 18 20:33:53.336637 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:33:53.336609    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383817fd_cd9b_4d6f_b3ab_338edc7f1e23.slice/crio-019e0d84882c78db57f306391ff20476d478ed8ca5f2893785cea2903017ab7a WatchSource:0}: Error finding container 019e0d84882c78db57f306391ff20476d478ed8ca5f2893785cea2903017ab7a: Status 404 returned error can't find the container with id 019e0d84882c78db57f306391ff20476d478ed8ca5f2893785cea2903017ab7a
Sep 18 20:33:53.968944 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:53.968909    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" event={"ID":"383817fd-cd9b-4d6f-b3ab-338edc7f1e23","Type":"ContainerStarted","Data":"46ea61d1bc8e46296d8893d9fcfb3efd1775ccbe360cb7545012bb2281f44e2a"}
Sep 18 20:33:53.969484 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:53.968969    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" event={"ID":"383817fd-cd9b-4d6f-b3ab-338edc7f1e23","Type":"ContainerStarted","Data":"019e0d84882c78db57f306391ff20476d478ed8ca5f2893785cea2903017ab7a"}
Sep 18 20:33:56.984728 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:56.984702    2583 generic.go:356] "Generic (PLEG): container finished" podID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerID="46ea61d1bc8e46296d8893d9fcfb3efd1775ccbe360cb7545012bb2281f44e2a" exitCode=0
Sep 18 20:33:56.985017 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:56.984770    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" event={"ID":"383817fd-cd9b-4d6f-b3ab-338edc7f1e23","Type":"ContainerDied","Data":"46ea61d1bc8e46296d8893d9fcfb3efd1775ccbe360cb7545012bb2281f44e2a"}
Sep 18 20:33:57.991649 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:57.991618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" event={"ID":"383817fd-cd9b-4d6f-b3ab-338edc7f1e23","Type":"ContainerStarted","Data":"f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade"}
Sep 18 20:33:57.992026 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:57.991678    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" event={"ID":"383817fd-cd9b-4d6f-b3ab-338edc7f1e23","Type":"ContainerStarted","Data":"f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923"}
Sep 18 20:33:57.992026 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:57.991889    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:58.013872 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:58.013826    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podStartSLOduration=2.442592438 podStartE2EDuration="6.013811004s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.571218566s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:33:58.011144971 +0000 UTC m=+788.919047249" watchObservedRunningTime="2026-09-18 20:33:58.013811004 +0000 UTC m=+788.921713283"
Sep 18 20:33:58.996763 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:58.996725    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:33:58.998322 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:33:58.998296    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:34:00.003180 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:34:00.003138    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:34:05.007783 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:34:05.007752    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:34:05.008258 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:34:05.008233    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:34:15.009021 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:34:15.008981    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:34:25.009160 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:34:25.009098    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:34:35.009034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:34:35.008994    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:34:45.008796 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:34:45.008755    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:34:55.009201 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:34:55.009154    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:35:03.235345 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.235306    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"]
Sep 18 20:35:03.240480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.240460    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.243076 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.243056    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 18 20:35:03.243220 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.243073    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 18 20:35:03.248318 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.248293    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"]
Sep 18 20:35:03.418188 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.418149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b94d45-8fbf-4140-88b1-1d1e931cab03-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.418188 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.418192    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b94d45-8fbf-4140-88b1-1d1e931cab03-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.418428 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.418279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zj6\" (UniqueName: \"kubernetes.io/projected/01b94d45-8fbf-4140-88b1-1d1e931cab03-kube-api-access-f9zj6\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.418428 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.418304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b94d45-8fbf-4140-88b1-1d1e931cab03-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.418428 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.418373    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b94d45-8fbf-4140-88b1-1d1e931cab03-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.518947 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.518848    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zj6\" (UniqueName: \"kubernetes.io/projected/01b94d45-8fbf-4140-88b1-1d1e931cab03-kube-api-access-f9zj6\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.518947 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.518883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b94d45-8fbf-4140-88b1-1d1e931cab03-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.518947 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.518908    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b94d45-8fbf-4140-88b1-1d1e931cab03-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.519268 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.519011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b94d45-8fbf-4140-88b1-1d1e931cab03-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.519268 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.519037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b94d45-8fbf-4140-88b1-1d1e931cab03-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.519376 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.519352    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b94d45-8fbf-4140-88b1-1d1e931cab03-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.519781 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.519760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b94d45-8fbf-4140-88b1-1d1e931cab03-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.519862 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.519781    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b94d45-8fbf-4140-88b1-1d1e931cab03-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.521662 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.521639    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b94d45-8fbf-4140-88b1-1d1e931cab03-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.527660 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.527633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zj6\" (UniqueName: \"kubernetes.io/projected/01b94d45-8fbf-4140-88b1-1d1e931cab03-kube-api-access-f9zj6\") pod \"isvc-autogluon-v1-predictor-574b5b7495-xqcwc\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.551286 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.551256    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:35:03.828149 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:03.823679    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"]
Sep 18 20:35:03.831270 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:35:03.831234    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b94d45_8fbf_4140_88b1_1d1e931cab03.slice/crio-118fffe23a20af2eb9e5f2e365bf447607a7cab242adc4882586bc2861a3e55f WatchSource:0}: Error finding container 118fffe23a20af2eb9e5f2e365bf447607a7cab242adc4882586bc2861a3e55f: Status 404 returned error can't find the container with id 118fffe23a20af2eb9e5f2e365bf447607a7cab242adc4882586bc2861a3e55f
Sep 18 20:35:04.313980 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:04.313945    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" event={"ID":"01b94d45-8fbf-4140-88b1-1d1e931cab03","Type":"ContainerStarted","Data":"9c02eee67b1f897dff55373158940389234cb00147aa4276e0ea4af0368857b1"}
Sep 18 20:35:04.314383 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:04.313999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" event={"ID":"01b94d45-8fbf-4140-88b1-1d1e931cab03","Type":"ContainerStarted","Data":"118fffe23a20af2eb9e5f2e365bf447607a7cab242adc4882586bc2861a3e55f"}
Sep 18 20:35:05.008795 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:05.008755    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:35:09.310566 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:09.310526    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:35:19.315754 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:19.315726    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:35:23.253546 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:23.253507    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"]
Sep 18 20:35:23.253930 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:23.253803    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container" containerID="cri-o://f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923" gracePeriod=30
Sep 18 20:35:23.253930 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:23.253846    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kube-rbac-proxy" containerID="cri-o://f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade" gracePeriod=30
Sep 18 20:35:23.401587 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:23.401556    2583 generic.go:356] "Generic (PLEG): container finished" podID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerID="f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade" exitCode=2
Sep 18 20:35:23.401711 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:23.401594    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" event={"ID":"383817fd-cd9b-4d6f-b3ab-338edc7f1e23","Type":"ContainerDied","Data":"f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade"}
Sep 18 20:35:25.004021 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:25.003979    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 18 20:35:29.294381 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.294357    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:35:29.334931 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.334908    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-proxy-tls\") pod \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") "
Sep 18 20:35:29.335074 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.334955    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kserve-provision-location\") pod \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") "
Sep 18 20:35:29.335074 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.334989    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp2wz\" (UniqueName: \"kubernetes.io/projected/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kube-api-access-pp2wz\") pod \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") "
Sep 18 20:35:29.335074 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.335026    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-cabundle-cert\") pod \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") "
Sep 18 20:35:29.335074 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.335070    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kube-rbac-proxy-sar-config\") pod \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\" (UID: \"383817fd-cd9b-4d6f-b3ab-338edc7f1e23\") "
Sep 18 20:35:29.335368 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.335340    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "383817fd-cd9b-4d6f-b3ab-338edc7f1e23" (UID: "383817fd-cd9b-4d6f-b3ab-338edc7f1e23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:35:29.335423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.335410    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "383817fd-cd9b-4d6f-b3ab-338edc7f1e23" (UID: "383817fd-cd9b-4d6f-b3ab-338edc7f1e23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:35:29.335511 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.335483    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "383817fd-cd9b-4d6f-b3ab-338edc7f1e23" (UID: "383817fd-cd9b-4d6f-b3ab-338edc7f1e23"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:35:29.337217 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.337186    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "383817fd-cd9b-4d6f-b3ab-338edc7f1e23" (UID: "383817fd-cd9b-4d6f-b3ab-338edc7f1e23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:35:29.337310 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.337214    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kube-api-access-pp2wz" (OuterVolumeSpecName: "kube-api-access-pp2wz") pod "383817fd-cd9b-4d6f-b3ab-338edc7f1e23" (UID: "383817fd-cd9b-4d6f-b3ab-338edc7f1e23"). InnerVolumeSpecName "kube-api-access-pp2wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:35:29.435898 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.435869    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:35:29.436151 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.436099    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:35:29.436277 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.436152    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp2wz\" (UniqueName: \"kubernetes.io/projected/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kube-api-access-pp2wz\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:35:29.436277 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.436169    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:35:29.436277 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.436184    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383817fd-cd9b-4d6f-b3ab-338edc7f1e23-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:35:29.437147 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.437107    2583 generic.go:356] "Generic (PLEG): container finished" podID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerID="f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923" exitCode=0
Sep 18 20:35:29.437233 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.437162    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" event={"ID":"383817fd-cd9b-4d6f-b3ab-338edc7f1e23","Type":"ContainerDied","Data":"f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923"}
Sep 18 20:35:29.437233 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.437193    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"
Sep 18 20:35:29.437297 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.437261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9" event={"ID":"383817fd-cd9b-4d6f-b3ab-338edc7f1e23","Type":"ContainerDied","Data":"019e0d84882c78db57f306391ff20476d478ed8ca5f2893785cea2903017ab7a"}
Sep 18 20:35:29.437329 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.437300    2583 scope.go:117] "RemoveContainer" containerID="f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade"
Sep 18 20:35:29.446000 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.445971    2583 scope.go:117] "RemoveContainer" containerID="f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923"
Sep 18 20:35:29.453562 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.453546    2583 scope.go:117] "RemoveContainer" containerID="46ea61d1bc8e46296d8893d9fcfb3efd1775ccbe360cb7545012bb2281f44e2a"
Sep 18 20:35:29.460276 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.460258    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"]
Sep 18 20:35:29.461349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.461333    2583 scope.go:117] "RemoveContainer" containerID="f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade"
Sep 18 20:35:29.461569 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:35:29.461552    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade\": container with ID starting with f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade not found: ID does not exist" containerID="f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade"
Sep 18 20:35:29.461627 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.461574    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade"} err="failed to get container status \"f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade\": rpc error: code = NotFound desc = could not find container \"f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade\": container with ID starting with f8b2de0dfbf4e46bf75cb9c2bf58fdfd67f5a07d9c8ab268ebb3bcdd7d634ade not found: ID does not exist"
Sep 18 20:35:29.461627 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.461589    2583 scope.go:117] "RemoveContainer" containerID="f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923"
Sep 18 20:35:29.461873 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:35:29.461851    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923\": container with ID starting with f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923 not found: ID does not exist" containerID="f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923"
Sep 18 20:35:29.462137 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.461962    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923"} err="failed to get container status \"f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923\": rpc error: code = NotFound desc = could not find container \"f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923\": container with ID starting with f33b91749cf321aefd7e41630c0caf461d1232d1d928fc0e836c931362f43923 not found: ID does not exist"
Sep 18 20:35:29.462137 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.461994    2583 scope.go:117] "RemoveContainer" containerID="46ea61d1bc8e46296d8893d9fcfb3efd1775ccbe360cb7545012bb2281f44e2a"
Sep 18 20:35:29.462571 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:35:29.462547    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ea61d1bc8e46296d8893d9fcfb3efd1775ccbe360cb7545012bb2281f44e2a\": container with ID starting with 46ea61d1bc8e46296d8893d9fcfb3efd1775ccbe360cb7545012bb2281f44e2a not found: ID does not exist" containerID="46ea61d1bc8e46296d8893d9fcfb3efd1775ccbe360cb7545012bb2281f44e2a"
Sep 18 20:35:29.462639 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.462578    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ea61d1bc8e46296d8893d9fcfb3efd1775ccbe360cb7545012bb2281f44e2a"} err="failed to get container status \"46ea61d1bc8e46296d8893d9fcfb3efd1775ccbe360cb7545012bb2281f44e2a\": rpc error: code = NotFound desc = could not find container \"46ea61d1bc8e46296d8893d9fcfb3efd1775ccbe360cb7545012bb2281f44e2a\": container with ID starting with 46ea61d1bc8e46296d8893d9fcfb3efd1775ccbe360cb7545012bb2281f44e2a not found: ID does not exist"
Sep 18 20:35:29.464166 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:29.464148    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8zfc9"]
Sep 18 20:35:31.317507 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:31.317430    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" path="/var/lib/kubelet/pods/383817fd-cd9b-4d6f-b3ab-338edc7f1e23/volumes"
Sep 18 20:35:33.457555 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:33.457515    2583 generic.go:356] "Generic (PLEG): container finished" podID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerID="9c02eee67b1f897dff55373158940389234cb00147aa4276e0ea4af0368857b1" exitCode=0
Sep 18 20:35:33.457555 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:33.457559    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" event={"ID":"01b94d45-8fbf-4140-88b1-1d1e931cab03","Type":"ContainerDied","Data":"9c02eee67b1f897dff55373158940389234cb00147aa4276e0ea4af0368857b1"}
Sep 18 20:35:33.458803 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:33.458786    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:35:51.271075 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:51.271050    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:35:51.273543 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:35:51.273520    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:37:37.105475 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:37:37.105391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" event={"ID":"01b94d45-8fbf-4140-88b1-1d1e931cab03","Type":"ContainerStarted","Data":"120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f"}
Sep 18 20:37:37.105475 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:37:37.105443    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" event={"ID":"01b94d45-8fbf-4140-88b1-1d1e931cab03","Type":"ContainerStarted","Data":"5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0"}
Sep 18 20:37:37.105887 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:37:37.105498    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:37:37.124886 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:37:37.124834    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" podStartSLOduration=1.543297309 podStartE2EDuration="2m34.124815814s" totalImagesPullingTime="2m3.209151602s" totalInitContainerRuntime="29.372366903s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:37:37.123103523 +0000 UTC m=+1008.031005823" watchObservedRunningTime="2026-09-18 20:37:37.124815814 +0000 UTC m=+1008.032718094"
Sep 18 20:37:38.109759 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:37:38.109721    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:37:38.111244 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:37:38.111216    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 20:37:39.114828 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:37:39.114788    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 20:37:44.119186 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:37:44.119157    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:37:44.119627 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:37:44.119599    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 20:37:54.121300 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:37:54.121271    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:38:09.543604 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:09.543574    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"]
Sep 18 20:38:09.544026 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:09.543819    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="kserve-container" containerID="cri-o://5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0" gracePeriod=30
Sep 18 20:38:09.544026 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:09.543849    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="kube-rbac-proxy" containerID="cri-o://120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f" gracePeriod=30
Sep 18 20:38:10.270763 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:10.270724    2583 generic.go:356] "Generic (PLEG): container finished" podID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerID="120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f" exitCode=2
Sep 18 20:38:10.270938 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:10.270799    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" event={"ID":"01b94d45-8fbf-4140-88b1-1d1e931cab03","Type":"ContainerDied","Data":"120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f"}
Sep 18 20:38:11.010537 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.010508    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:38:11.096917 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.096835    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b94d45-8fbf-4140-88b1-1d1e931cab03-kserve-provision-location\") pod \"01b94d45-8fbf-4140-88b1-1d1e931cab03\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") "
Sep 18 20:38:11.096917 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.096903    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b94d45-8fbf-4140-88b1-1d1e931cab03-proxy-tls\") pod \"01b94d45-8fbf-4140-88b1-1d1e931cab03\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") "
Sep 18 20:38:11.097214 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.096950    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b94d45-8fbf-4140-88b1-1d1e931cab03-kube-rbac-proxy-sar-config\") pod \"01b94d45-8fbf-4140-88b1-1d1e931cab03\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") "
Sep 18 20:38:11.097214 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.096976    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b94d45-8fbf-4140-88b1-1d1e931cab03-cabundle-cert\") pod \"01b94d45-8fbf-4140-88b1-1d1e931cab03\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") "
Sep 18 20:38:11.097214 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.097032    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9zj6\" (UniqueName: \"kubernetes.io/projected/01b94d45-8fbf-4140-88b1-1d1e931cab03-kube-api-access-f9zj6\") pod \"01b94d45-8fbf-4140-88b1-1d1e931cab03\" (UID: \"01b94d45-8fbf-4140-88b1-1d1e931cab03\") "
Sep 18 20:38:11.097398 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.097373    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b94d45-8fbf-4140-88b1-1d1e931cab03-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "01b94d45-8fbf-4140-88b1-1d1e931cab03" (UID: "01b94d45-8fbf-4140-88b1-1d1e931cab03"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:38:11.097398 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.097382    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b94d45-8fbf-4140-88b1-1d1e931cab03-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "01b94d45-8fbf-4140-88b1-1d1e931cab03" (UID: "01b94d45-8fbf-4140-88b1-1d1e931cab03"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:38:11.099313 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.099285    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b94d45-8fbf-4140-88b1-1d1e931cab03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01b94d45-8fbf-4140-88b1-1d1e931cab03" (UID: "01b94d45-8fbf-4140-88b1-1d1e931cab03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:38:11.099448 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.099363    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b94d45-8fbf-4140-88b1-1d1e931cab03-kube-api-access-f9zj6" (OuterVolumeSpecName: "kube-api-access-f9zj6") pod "01b94d45-8fbf-4140-88b1-1d1e931cab03" (UID: "01b94d45-8fbf-4140-88b1-1d1e931cab03"). InnerVolumeSpecName "kube-api-access-f9zj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:38:11.101601 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.101573    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b94d45-8fbf-4140-88b1-1d1e931cab03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01b94d45-8fbf-4140-88b1-1d1e931cab03" (UID: "01b94d45-8fbf-4140-88b1-1d1e931cab03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:38:11.198459 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.198427    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b94d45-8fbf-4140-88b1-1d1e931cab03-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:38:11.198611 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.198471    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9zj6\" (UniqueName: \"kubernetes.io/projected/01b94d45-8fbf-4140-88b1-1d1e931cab03-kube-api-access-f9zj6\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:38:11.198611 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.198480    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b94d45-8fbf-4140-88b1-1d1e931cab03-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:38:11.198611 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.198489    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b94d45-8fbf-4140-88b1-1d1e931cab03-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:38:11.198611 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.198499    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b94d45-8fbf-4140-88b1-1d1e931cab03-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:38:11.277618 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.277587    2583 generic.go:356] "Generic (PLEG): container finished" podID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerID="5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0" exitCode=0
Sep 18 20:38:11.277767 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.277626    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" event={"ID":"01b94d45-8fbf-4140-88b1-1d1e931cab03","Type":"ContainerDied","Data":"5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0"}
Sep 18 20:38:11.277767 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.277648    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"
Sep 18 20:38:11.277767 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.277672    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc" event={"ID":"01b94d45-8fbf-4140-88b1-1d1e931cab03","Type":"ContainerDied","Data":"118fffe23a20af2eb9e5f2e365bf447607a7cab242adc4882586bc2861a3e55f"}
Sep 18 20:38:11.277767 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.277698    2583 scope.go:117] "RemoveContainer" containerID="120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f"
Sep 18 20:38:11.287023 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.287006    2583 scope.go:117] "RemoveContainer" containerID="5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0"
Sep 18 20:38:11.294827 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.294807    2583 scope.go:117] "RemoveContainer" containerID="9c02eee67b1f897dff55373158940389234cb00147aa4276e0ea4af0368857b1"
Sep 18 20:38:11.301436 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.301414    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"]
Sep 18 20:38:11.303075 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.303013    2583 scope.go:117] "RemoveContainer" containerID="120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f"
Sep 18 20:38:11.303521 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:38:11.303497    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f\": container with ID starting with 120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f not found: ID does not exist" containerID="120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f"
Sep 18 20:38:11.303611 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.303531    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f"} err="failed to get container status \"120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f\": rpc error: code = NotFound desc = could not find container \"120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f\": container with ID starting with 120a128b13a533abd2f7893a7a98369bee608e497fba1015617919c0d2d6c93f not found: ID does not exist"
Sep 18 20:38:11.303611 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.303552    2583 scope.go:117] "RemoveContainer" containerID="5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0"
Sep 18 20:38:11.303839 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:38:11.303817    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0\": container with ID starting with 5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0 not found: ID does not exist" containerID="5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0"
Sep 18 20:38:11.303910 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.303849    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0"} err="failed to get container status \"5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0\": rpc error: code = NotFound desc = could not find container \"5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0\": container with ID starting with 5c57967b279e0265e86910ec32859e2b9a1f3ba1a2547fe20d5a30ee549187c0 not found: ID does not exist"
Sep 18 20:38:11.303910 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.303873    2583 scope.go:117] "RemoveContainer" containerID="9c02eee67b1f897dff55373158940389234cb00147aa4276e0ea4af0368857b1"
Sep 18 20:38:11.304230 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:38:11.304205    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c02eee67b1f897dff55373158940389234cb00147aa4276e0ea4af0368857b1\": container with ID starting with 9c02eee67b1f897dff55373158940389234cb00147aa4276e0ea4af0368857b1 not found: ID does not exist" containerID="9c02eee67b1f897dff55373158940389234cb00147aa4276e0ea4af0368857b1"
Sep 18 20:38:11.304322 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.304236    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c02eee67b1f897dff55373158940389234cb00147aa4276e0ea4af0368857b1"} err="failed to get container status \"9c02eee67b1f897dff55373158940389234cb00147aa4276e0ea4af0368857b1\": rpc error: code = NotFound desc = could not find container \"9c02eee67b1f897dff55373158940389234cb00147aa4276e0ea4af0368857b1\": container with ID starting with 9c02eee67b1f897dff55373158940389234cb00147aa4276e0ea4af0368857b1 not found: ID does not exist"
Sep 18 20:38:11.305083 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.305065    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-xqcwc"]
Sep 18 20:38:11.316222 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:11.316201    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" path="/var/lib/kubelet/pods/01b94d45-8fbf-4140-88b1-1d1e931cab03/volumes"
Sep 18 20:38:14.739461 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739422    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"]
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739840    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="storage-initializer"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739855    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="storage-initializer"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739865    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="kserve-container"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739871    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="kserve-container"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739882    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="storage-initializer"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739888    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="storage-initializer"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739895    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="kube-rbac-proxy"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739900    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="kube-rbac-proxy"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739910    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739919    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739930    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kube-rbac-proxy"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739935    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kube-rbac-proxy"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.739993    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="kube-rbac-proxy"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.740004    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b94d45-8fbf-4140-88b1-1d1e931cab03" containerName="kserve-container"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.740012    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kserve-container"
Sep 18 20:38:14.740093 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.740018    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="383817fd-cd9b-4d6f-b3ab-338edc7f1e23" containerName="kube-rbac-proxy"
Sep 18 20:38:14.743579 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.743560    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.749169 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.749143    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:38:14.749352 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.749319    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 18 20:38:14.749468 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.749225    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:38:14.749468 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.749236    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:38:14.749468 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.749283    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:38:14.749670 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.749210    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 18 20:38:14.750558 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.750246    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:38:14.754974 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.754952    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"]
Sep 18 20:38:14.824276 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.824243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zgc\" (UniqueName: \"kubernetes.io/projected/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kube-api-access-g9zgc\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.824435 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.824305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.824435 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.824361    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfcc1a39-8509-404d-bb8a-e86b43b1b789-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.824435 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.824422    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfcc1a39-8509-404d-bb8a-e86b43b1b789-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.824540 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.824440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.925178 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.925146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zgc\" (UniqueName: \"kubernetes.io/projected/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kube-api-access-g9zgc\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.925321 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.925202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.925321 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.925242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfcc1a39-8509-404d-bb8a-e86b43b1b789-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.925432 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.925373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfcc1a39-8509-404d-bb8a-e86b43b1b789-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.925432 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.925413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.925518 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:38:14.925498    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 18 20:38:14.925586 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:38:14.925573    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfcc1a39-8509-404d-bb8a-e86b43b1b789-proxy-tls podName:dfcc1a39-8509-404d-bb8a-e86b43b1b789 nodeName:}" failed. No retries permitted until 2026-09-18 20:38:15.425554882 +0000 UTC m=+1046.333457138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dfcc1a39-8509-404d-bb8a-e86b43b1b789-proxy-tls") pod "isvc-autogluon-v2-predictor-9684b4cd9-97mrp" (UID: "dfcc1a39-8509-404d-bb8a-e86b43b1b789") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 18 20:38:14.925755 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.925737    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.925876 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.925861    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfcc1a39-8509-404d-bb8a-e86b43b1b789-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.925923 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.925909    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:14.936734 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:14.936709    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zgc\" (UniqueName: \"kubernetes.io/projected/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kube-api-access-g9zgc\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:15.429054 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:15.429020    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfcc1a39-8509-404d-bb8a-e86b43b1b789-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:15.431580 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:15.431553    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfcc1a39-8509-404d-bb8a-e86b43b1b789-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-97mrp\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:15.654760 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:15.654718    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:15.926954 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:15.926925    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"]
Sep 18 20:38:15.928803 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:38:15.928774    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcc1a39_8509_404d_bb8a_e86b43b1b789.slice/crio-593ab9f384876f6ce9b0326f66c896915eecb6e075ee6ef7990ee6c0e625e76b WatchSource:0}: Error finding container 593ab9f384876f6ce9b0326f66c896915eecb6e075ee6ef7990ee6c0e625e76b: Status 404 returned error can't find the container with id 593ab9f384876f6ce9b0326f66c896915eecb6e075ee6ef7990ee6c0e625e76b
Sep 18 20:38:16.304264 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:16.304180    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp" event={"ID":"dfcc1a39-8509-404d-bb8a-e86b43b1b789","Type":"ContainerStarted","Data":"48d813322498c408b591e70aa090a3473d5f6dc43520a142431d81d551238088"}
Sep 18 20:38:16.304264 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:16.304252    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp" event={"ID":"dfcc1a39-8509-404d-bb8a-e86b43b1b789","Type":"ContainerStarted","Data":"593ab9f384876f6ce9b0326f66c896915eecb6e075ee6ef7990ee6c0e625e76b"}
Sep 18 20:38:46.443627 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:46.443594    2583 generic.go:356] "Generic (PLEG): container finished" podID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerID="48d813322498c408b591e70aa090a3473d5f6dc43520a142431d81d551238088" exitCode=0
Sep 18 20:38:46.444157 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:46.443669    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp" event={"ID":"dfcc1a39-8509-404d-bb8a-e86b43b1b789","Type":"ContainerDied","Data":"48d813322498c408b591e70aa090a3473d5f6dc43520a142431d81d551238088"}
Sep 18 20:38:47.451057 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:47.451026    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp" event={"ID":"dfcc1a39-8509-404d-bb8a-e86b43b1b789","Type":"ContainerStarted","Data":"01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530"}
Sep 18 20:38:47.451483 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:47.451082    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp" event={"ID":"dfcc1a39-8509-404d-bb8a-e86b43b1b789","Type":"ContainerStarted","Data":"839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096"}
Sep 18 20:38:47.451483 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:47.451300    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:47.471342 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:47.471290    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp" podStartSLOduration=3.925692001 podStartE2EDuration="33.471269447s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.545577446s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:38:47.469400369 +0000 UTC m=+1078.377302644" watchObservedRunningTime="2026-09-18 20:38:47.471269447 +0000 UTC m=+1078.379171727"
Sep 18 20:38:48.456284 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:48.456254    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:38:54.471184 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:38:54.471150    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:40:21.347619 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:21.347583    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:40:30.994066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:30.994033    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"]
Sep 18 20:40:30.994558 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:30.994296    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp" podUID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerName="kserve-container" containerID="cri-o://839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096" gracePeriod=30
Sep 18 20:40:30.994558 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:30.994326    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp" podUID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerName="kube-rbac-proxy" containerID="cri-o://01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530" gracePeriod=30
Sep 18 20:40:31.313500 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:31.313415    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp" podUID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-autogluon-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:40:31.945607 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:31.945574    2583 generic.go:356] "Generic (PLEG): container finished" podID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerID="01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530" exitCode=2
Sep 18 20:40:31.945790 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:31.945641    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp" event={"ID":"dfcc1a39-8509-404d-bb8a-e86b43b1b789","Type":"ContainerDied","Data":"01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530"}
Sep 18 20:40:32.438776 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.438746    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:40:32.562403 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.562335    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfcc1a39-8509-404d-bb8a-e86b43b1b789-proxy-tls\") pod \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") "
Sep 18 20:40:32.562403 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.562376    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfcc1a39-8509-404d-bb8a-e86b43b1b789-cabundle-cert\") pod \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") "
Sep 18 20:40:32.562612 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.562420    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kube-rbac-proxy-sar-config\") pod \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") "
Sep 18 20:40:32.562612 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.562514    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9zgc\" (UniqueName: \"kubernetes.io/projected/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kube-api-access-g9zgc\") pod \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") "
Sep 18 20:40:32.562612 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.562553    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kserve-provision-location\") pod \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\" (UID: \"dfcc1a39-8509-404d-bb8a-e86b43b1b789\") "
Sep 18 20:40:32.562769 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.562748    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dfcc1a39-8509-404d-bb8a-e86b43b1b789" (UID: "dfcc1a39-8509-404d-bb8a-e86b43b1b789"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:40:32.562818 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.562768    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcc1a39-8509-404d-bb8a-e86b43b1b789-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dfcc1a39-8509-404d-bb8a-e86b43b1b789" (UID: "dfcc1a39-8509-404d-bb8a-e86b43b1b789"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:40:32.564613 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.564580    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfcc1a39-8509-404d-bb8a-e86b43b1b789-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dfcc1a39-8509-404d-bb8a-e86b43b1b789" (UID: "dfcc1a39-8509-404d-bb8a-e86b43b1b789"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:40:32.564722 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.564688    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kube-api-access-g9zgc" (OuterVolumeSpecName: "kube-api-access-g9zgc") pod "dfcc1a39-8509-404d-bb8a-e86b43b1b789" (UID: "dfcc1a39-8509-404d-bb8a-e86b43b1b789"). InnerVolumeSpecName "kube-api-access-g9zgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:40:32.566569 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.566547    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dfcc1a39-8509-404d-bb8a-e86b43b1b789" (UID: "dfcc1a39-8509-404d-bb8a-e86b43b1b789"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:40:32.664049 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.664021    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:40:32.664049 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.664045    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9zgc\" (UniqueName: \"kubernetes.io/projected/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kube-api-access-g9zgc\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:40:32.664049 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.664056    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfcc1a39-8509-404d-bb8a-e86b43b1b789-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:40:32.664293 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.664065    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfcc1a39-8509-404d-bb8a-e86b43b1b789-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:40:32.664293 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.664075    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfcc1a39-8509-404d-bb8a-e86b43b1b789-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:40:32.952090 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.952061    2583 generic.go:356] "Generic (PLEG): container finished" podID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerID="839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096" exitCode=0
Sep 18 20:40:32.952256 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.952145    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"
Sep 18 20:40:32.952256 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.952143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp" event={"ID":"dfcc1a39-8509-404d-bb8a-e86b43b1b789","Type":"ContainerDied","Data":"839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096"}
Sep 18 20:40:32.952331 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.952266    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp" event={"ID":"dfcc1a39-8509-404d-bb8a-e86b43b1b789","Type":"ContainerDied","Data":"593ab9f384876f6ce9b0326f66c896915eecb6e075ee6ef7990ee6c0e625e76b"}
Sep 18 20:40:32.952331 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.952293    2583 scope.go:117] "RemoveContainer" containerID="01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530"
Sep 18 20:40:32.961377 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.961360    2583 scope.go:117] "RemoveContainer" containerID="839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096"
Sep 18 20:40:32.970769 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.970731    2583 scope.go:117] "RemoveContainer" containerID="48d813322498c408b591e70aa090a3473d5f6dc43520a142431d81d551238088"
Sep 18 20:40:32.974116 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.974095    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"]
Sep 18 20:40:32.978513 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.978490    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-97mrp"]
Sep 18 20:40:32.979892 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.979878    2583 scope.go:117] "RemoveContainer" containerID="01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530"
Sep 18 20:40:32.980166 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:40:32.980149    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530\": container with ID starting with 01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530 not found: ID does not exist" containerID="01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530"
Sep 18 20:40:32.980217 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.980176    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530"} err="failed to get container status \"01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530\": rpc error: code = NotFound desc = could not find container \"01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530\": container with ID starting with 01259e47bbd5c8a539d1c1eb586eb2571ccd06108ca14a96ec698930820e8530 not found: ID does not exist"
Sep 18 20:40:32.980217 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.980193    2583 scope.go:117] "RemoveContainer" containerID="839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096"
Sep 18 20:40:32.980437 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:40:32.980417    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096\": container with ID starting with 839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096 not found: ID does not exist" containerID="839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096"
Sep 18 20:40:32.980480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.980444    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096"} err="failed to get container status \"839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096\": rpc error: code = NotFound desc = could not find container \"839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096\": container with ID starting with 839e035b403170619630056bdbb1a74f9d4e0fe36f16df698852de012b0c6096 not found: ID does not exist"
Sep 18 20:40:32.980480 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.980462    2583 scope.go:117] "RemoveContainer" containerID="48d813322498c408b591e70aa090a3473d5f6dc43520a142431d81d551238088"
Sep 18 20:40:32.980701 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:40:32.980685    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d813322498c408b591e70aa090a3473d5f6dc43520a142431d81d551238088\": container with ID starting with 48d813322498c408b591e70aa090a3473d5f6dc43520a142431d81d551238088 not found: ID does not exist" containerID="48d813322498c408b591e70aa090a3473d5f6dc43520a142431d81d551238088"
Sep 18 20:40:32.980743 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:32.980703    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d813322498c408b591e70aa090a3473d5f6dc43520a142431d81d551238088"} err="failed to get container status \"48d813322498c408b591e70aa090a3473d5f6dc43520a142431d81d551238088\": rpc error: code = NotFound desc = could not find container \"48d813322498c408b591e70aa090a3473d5f6dc43520a142431d81d551238088\": container with ID starting with 48d813322498c408b591e70aa090a3473d5f6dc43520a142431d81d551238088 not found: ID does not exist"
Sep 18 20:40:33.316337 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:33.316246    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" path="/var/lib/kubelet/pods/dfcc1a39-8509-404d-bb8a-e86b43b1b789/volumes"
Sep 18 20:40:51.300768 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:51.300739    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:40:51.304216 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:40:51.304198    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:41:00.051427 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.051385    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-e3b72aa1"]
Sep 18 20:41:00.051804 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.051725    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerName="storage-initializer"
Sep 18 20:41:00.051804 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.051737    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerName="storage-initializer"
Sep 18 20:41:00.051804 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.051752    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerName="kserve-container"
Sep 18 20:41:00.051804 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.051757    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerName="kserve-container"
Sep 18 20:41:00.051804 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.051774    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerName="kube-rbac-proxy"
Sep 18 20:41:00.051804 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.051780    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerName="kube-rbac-proxy"
Sep 18 20:41:00.051995 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.051836    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerName="kube-rbac-proxy"
Sep 18 20:41:00.051995 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.051846    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfcc1a39-8509-404d-bb8a-e86b43b1b789" containerName="kserve-container"
Sep 18 20:41:00.054328 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.054312    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-e3b72aa1"
Sep 18 20:41:00.058738 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.058712    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-e3b72aa1"]
Sep 18 20:41:00.198169 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.198118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96p74\" (UniqueName: \"kubernetes.io/projected/144b79ec-9429-4a3d-9ef1-fdfbec8a296b-kube-api-access-96p74\") pod \"curl-metrics-e3b72aa1\" (UID: \"144b79ec-9429-4a3d-9ef1-fdfbec8a296b\") " pod="kserve/curl-metrics-e3b72aa1"
Sep 18 20:41:00.298774 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.298742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96p74\" (UniqueName: \"kubernetes.io/projected/144b79ec-9429-4a3d-9ef1-fdfbec8a296b-kube-api-access-96p74\") pod \"curl-metrics-e3b72aa1\" (UID: \"144b79ec-9429-4a3d-9ef1-fdfbec8a296b\") " pod="kserve/curl-metrics-e3b72aa1"
Sep 18 20:41:00.308115 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.308053    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96p74\" (UniqueName: \"kubernetes.io/projected/144b79ec-9429-4a3d-9ef1-fdfbec8a296b-kube-api-access-96p74\") pod \"curl-metrics-e3b72aa1\" (UID: \"144b79ec-9429-4a3d-9ef1-fdfbec8a296b\") " pod="kserve/curl-metrics-e3b72aa1"
Sep 18 20:41:00.373778 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.373753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-e3b72aa1"
Sep 18 20:41:00.641904 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.641882    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-e3b72aa1"]
Sep 18 20:41:00.643927 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:41:00.643898    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod144b79ec_9429_4a3d_9ef1_fdfbec8a296b.slice/crio-34e3f638d3ac42b7e05537dc20169a0ba3d1af8b0cd80bb1c34bb0edb7cee088 WatchSource:0}: Error finding container 34e3f638d3ac42b7e05537dc20169a0ba3d1af8b0cd80bb1c34bb0edb7cee088: Status 404 returned error can't find the container with id 34e3f638d3ac42b7e05537dc20169a0ba3d1af8b0cd80bb1c34bb0edb7cee088
Sep 18 20:41:00.645837 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:00.645821    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:41:01.078919 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:01.078883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-e3b72aa1" event={"ID":"144b79ec-9429-4a3d-9ef1-fdfbec8a296b","Type":"ContainerStarted","Data":"34e3f638d3ac42b7e05537dc20169a0ba3d1af8b0cd80bb1c34bb0edb7cee088"}
Sep 18 20:41:02.085147 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:02.085084    2583 generic.go:356] "Generic (PLEG): container finished" podID="144b79ec-9429-4a3d-9ef1-fdfbec8a296b" containerID="8d1042b3fc41a6ed72243f94671fa920e11988cb810675c355976a8f0823d95f" exitCode=0
Sep 18 20:41:02.085573 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:02.085174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-e3b72aa1" event={"ID":"144b79ec-9429-4a3d-9ef1-fdfbec8a296b","Type":"ContainerDied","Data":"8d1042b3fc41a6ed72243f94671fa920e11988cb810675c355976a8f0823d95f"}
Sep 18 20:41:03.215363 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:03.215340    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-e3b72aa1"
Sep 18 20:41:03.250215 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:03.250186    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-e3b72aa1_144b79ec-9429-4a3d-9ef1-fdfbec8a296b/curl-metrics-e3b72aa1/0.log"
Sep 18 20:41:03.275666 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:03.275639    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-e3b72aa1"]
Sep 18 20:41:03.279501 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:03.279480    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-e3b72aa1"]
Sep 18 20:41:03.326060 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:03.326037    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96p74\" (UniqueName: \"kubernetes.io/projected/144b79ec-9429-4a3d-9ef1-fdfbec8a296b-kube-api-access-96p74\") pod \"144b79ec-9429-4a3d-9ef1-fdfbec8a296b\" (UID: \"144b79ec-9429-4a3d-9ef1-fdfbec8a296b\") "
Sep 18 20:41:03.328191 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:03.328170    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144b79ec-9429-4a3d-9ef1-fdfbec8a296b-kube-api-access-96p74" (OuterVolumeSpecName: "kube-api-access-96p74") pod "144b79ec-9429-4a3d-9ef1-fdfbec8a296b" (UID: "144b79ec-9429-4a3d-9ef1-fdfbec8a296b"). InnerVolumeSpecName "kube-api-access-96p74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:41:03.427527 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:03.427500    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96p74\" (UniqueName: \"kubernetes.io/projected/144b79ec-9429-4a3d-9ef1-fdfbec8a296b-kube-api-access-96p74\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:41:04.095677 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:04.095650    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-e3b72aa1"
Sep 18 20:41:04.095677 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:04.095668    2583 scope.go:117] "RemoveContainer" containerID="8d1042b3fc41a6ed72243f94671fa920e11988cb810675c355976a8f0823d95f"
Sep 18 20:41:05.315851 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:41:05.315821    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144b79ec-9429-4a3d-9ef1-fdfbec8a296b" path="/var/lib/kubelet/pods/144b79ec-9429-4a3d-9ef1-fdfbec8a296b/volumes"
Sep 18 20:43:24.641794 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.641760    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"]
Sep 18 20:43:24.642184 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.642172    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144b79ec-9429-4a3d-9ef1-fdfbec8a296b" containerName="curl-metrics-e3b72aa1"
Sep 18 20:43:24.642184 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.642183    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="144b79ec-9429-4a3d-9ef1-fdfbec8a296b" containerName="curl-metrics-e3b72aa1"
Sep 18 20:43:24.642258 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.642244    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="144b79ec-9429-4a3d-9ef1-fdfbec8a296b" containerName="curl-metrics-e3b72aa1"
Sep 18 20:43:24.645398 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.645383    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.648115 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.648087    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 18 20:43:24.648115 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.648107    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 18 20:43:24.648115 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.648111    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:43:24.648457 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.648174    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:43:24.648457 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.648275    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:43:24.648457 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.648405    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:43:24.648629 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.648497    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:43:24.655681 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.655658    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"]
Sep 18 20:43:24.705901 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.705871    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbcdcafb-2285-4936-aa51-4a84c387f65d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.706008 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.705973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbcdcafb-2285-4936-aa51-4a84c387f65d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.706051 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.706035    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-778vx\" (UniqueName: \"kubernetes.io/projected/cbcdcafb-2285-4936-aa51-4a84c387f65d-kube-api-access-778vx\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.706088 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.706071    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbcdcafb-2285-4936-aa51-4a84c387f65d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.706169 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.706091    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbcdcafb-2285-4936-aa51-4a84c387f65d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.807335 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.807306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbcdcafb-2285-4936-aa51-4a84c387f65d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.807335 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.807338    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbcdcafb-2285-4936-aa51-4a84c387f65d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.807561 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.807359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbcdcafb-2285-4936-aa51-4a84c387f65d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.807561 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.807418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbcdcafb-2285-4936-aa51-4a84c387f65d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.807561 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.807446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-778vx\" (UniqueName: \"kubernetes.io/projected/cbcdcafb-2285-4936-aa51-4a84c387f65d-kube-api-access-778vx\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.807743 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:43:24.807553    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 18 20:43:24.807743 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:43:24.807627    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbcdcafb-2285-4936-aa51-4a84c387f65d-proxy-tls podName:cbcdcafb-2285-4936-aa51-4a84c387f65d nodeName:}" failed. No retries permitted until 2026-09-18 20:43:25.307607026 +0000 UTC m=+1356.215509286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cbcdcafb-2285-4936-aa51-4a84c387f65d-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" (UID: "cbcdcafb-2285-4936-aa51-4a84c387f65d") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 18 20:43:24.807891 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.807797    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbcdcafb-2285-4936-aa51-4a84c387f65d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.808082 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.808062    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbcdcafb-2285-4936-aa51-4a84c387f65d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.808181 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.808155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbcdcafb-2285-4936-aa51-4a84c387f65d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:24.821347 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:24.821327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-778vx\" (UniqueName: \"kubernetes.io/projected/cbcdcafb-2285-4936-aa51-4a84c387f65d-kube-api-access-778vx\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:25.310398 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:25.310366    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbcdcafb-2285-4936-aa51-4a84c387f65d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:25.313048 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:25.313020    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbcdcafb-2285-4936-aa51-4a84c387f65d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:25.556698 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:25.556665    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:43:25.827961 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:25.827913    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"]
Sep 18 20:43:25.830439 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:43:25.830410    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcdcafb_2285_4936_aa51_4a84c387f65d.slice/crio-9bf2e991657c15e1b6e36f925f3f715765037c35191e4ad861a1737368a4ec32 WatchSource:0}: Error finding container 9bf2e991657c15e1b6e36f925f3f715765037c35191e4ad861a1737368a4ec32: Status 404 returned error can't find the container with id 9bf2e991657c15e1b6e36f925f3f715765037c35191e4ad861a1737368a4ec32
Sep 18 20:43:26.743681 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:26.743650    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" event={"ID":"cbcdcafb-2285-4936-aa51-4a84c387f65d","Type":"ContainerStarted","Data":"468274e6a51f888ffb509d897603f0a9bcf8e47772a79cf0ead8a206a8e51708"}
Sep 18 20:43:26.743870 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:26.743705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" event={"ID":"cbcdcafb-2285-4936-aa51-4a84c387f65d","Type":"ContainerStarted","Data":"9bf2e991657c15e1b6e36f925f3f715765037c35191e4ad861a1737368a4ec32"}
Sep 18 20:43:29.763090 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:29.763057    2583 generic.go:356] "Generic (PLEG): container finished" podID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerID="468274e6a51f888ffb509d897603f0a9bcf8e47772a79cf0ead8a206a8e51708" exitCode=0
Sep 18 20:43:29.763495 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:43:29.763134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" event={"ID":"cbcdcafb-2285-4936-aa51-4a84c387f65d","Type":"ContainerDied","Data":"468274e6a51f888ffb509d897603f0a9bcf8e47772a79cf0ead8a206a8e51708"}
Sep 18 20:45:31.432348 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:45:31.432308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" event={"ID":"cbcdcafb-2285-4936-aa51-4a84c387f65d","Type":"ContainerStarted","Data":"c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa"}
Sep 18 20:45:31.432844 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:45:31.432366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" event={"ID":"cbcdcafb-2285-4936-aa51-4a84c387f65d","Type":"ContainerStarted","Data":"4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c"}
Sep 18 20:45:31.432844 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:45:31.432533    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:45:31.454263 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:45:31.454190    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" podStartSLOduration=2.79483864 podStartE2EDuration="2m7.454170919s" totalImagesPullingTime="2m1.026546657s" totalInitContainerRuntime="3.632785622s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:45:31.453353303 +0000 UTC m=+1482.361255586" watchObservedRunningTime="2026-09-18 20:45:31.454170919 +0000 UTC m=+1482.362073199"
Sep 18 20:45:32.437926 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:45:32.437891    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:45:38.449257 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:45:38.449230    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:45:51.341492 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:45:51.341466    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:45:51.341918 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:45:51.341472    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:46:08.524704 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:08.524602    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 18 20:46:18.452025 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:18.451949    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:46:24.769585 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:24.769551    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"]
Sep 18 20:46:24.769981 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:24.769780    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="kserve-container" containerID="cri-o://4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c" gracePeriod=30
Sep 18 20:46:24.769981 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:24.769862    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="kube-rbac-proxy" containerID="cri-o://c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa" gracePeriod=30
Sep 18 20:46:25.692379 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:25.692344    2583 generic.go:356] "Generic (PLEG): container finished" podID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerID="c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa" exitCode=2
Sep 18 20:46:25.692566 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:25.692407    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" event={"ID":"cbcdcafb-2285-4936-aa51-4a84c387f65d","Type":"ContainerDied","Data":"c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa"}
Sep 18 20:46:28.443653 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:28.443616    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 18 20:46:29.490291 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:29.490246    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 20:46:33.443436 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:33.443391    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 18 20:46:35.052074 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.052050    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:46:35.153283 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.153191    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-778vx\" (UniqueName: \"kubernetes.io/projected/cbcdcafb-2285-4936-aa51-4a84c387f65d-kube-api-access-778vx\") pod \"cbcdcafb-2285-4936-aa51-4a84c387f65d\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") "
Sep 18 20:46:35.153283 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.153247    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbcdcafb-2285-4936-aa51-4a84c387f65d-proxy-tls\") pod \"cbcdcafb-2285-4936-aa51-4a84c387f65d\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") "
Sep 18 20:46:35.153527 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.153304    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbcdcafb-2285-4936-aa51-4a84c387f65d-kube-rbac-proxy-sar-config\") pod \"cbcdcafb-2285-4936-aa51-4a84c387f65d\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") "
Sep 18 20:46:35.153527 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.153428    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbcdcafb-2285-4936-aa51-4a84c387f65d-kserve-provision-location\") pod \"cbcdcafb-2285-4936-aa51-4a84c387f65d\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") "
Sep 18 20:46:35.153527 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.153476    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbcdcafb-2285-4936-aa51-4a84c387f65d-cabundle-cert\") pod \"cbcdcafb-2285-4936-aa51-4a84c387f65d\" (UID: \"cbcdcafb-2285-4936-aa51-4a84c387f65d\") "
Sep 18 20:46:35.153765 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.153741    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcdcafb-2285-4936-aa51-4a84c387f65d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cbcdcafb-2285-4936-aa51-4a84c387f65d" (UID: "cbcdcafb-2285-4936-aa51-4a84c387f65d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:46:35.153827 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.153758    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcdcafb-2285-4936-aa51-4a84c387f65d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cbcdcafb-2285-4936-aa51-4a84c387f65d" (UID: "cbcdcafb-2285-4936-aa51-4a84c387f65d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:46:35.153868 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.153839    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcdcafb-2285-4936-aa51-4a84c387f65d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cbcdcafb-2285-4936-aa51-4a84c387f65d" (UID: "cbcdcafb-2285-4936-aa51-4a84c387f65d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:46:35.156034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.156004    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbcdcafb-2285-4936-aa51-4a84c387f65d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cbcdcafb-2285-4936-aa51-4a84c387f65d" (UID: "cbcdcafb-2285-4936-aa51-4a84c387f65d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:46:35.156034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.156016    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcdcafb-2285-4936-aa51-4a84c387f65d-kube-api-access-778vx" (OuterVolumeSpecName: "kube-api-access-778vx") pod "cbcdcafb-2285-4936-aa51-4a84c387f65d" (UID: "cbcdcafb-2285-4936-aa51-4a84c387f65d"). InnerVolumeSpecName "kube-api-access-778vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:46:35.255014 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.254964    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbcdcafb-2285-4936-aa51-4a84c387f65d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:46:35.255014 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.255006    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbcdcafb-2285-4936-aa51-4a84c387f65d-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:46:35.255014 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.255016    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbcdcafb-2285-4936-aa51-4a84c387f65d-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:46:35.255014 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.255024    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-778vx\" (UniqueName: \"kubernetes.io/projected/cbcdcafb-2285-4936-aa51-4a84c387f65d-kube-api-access-778vx\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:46:35.255320 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.255036    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbcdcafb-2285-4936-aa51-4a84c387f65d-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:46:35.740115 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.740080    2583 generic.go:356] "Generic (PLEG): container finished" podID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerID="4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c" exitCode=0
Sep 18 20:46:35.740337 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.740158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" event={"ID":"cbcdcafb-2285-4936-aa51-4a84c387f65d","Type":"ContainerDied","Data":"4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c"}
Sep 18 20:46:35.740337 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.740170    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"
Sep 18 20:46:35.740337 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.740227    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f" event={"ID":"cbcdcafb-2285-4936-aa51-4a84c387f65d","Type":"ContainerDied","Data":"9bf2e991657c15e1b6e36f925f3f715765037c35191e4ad861a1737368a4ec32"}
Sep 18 20:46:35.740337 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.740269    2583 scope.go:117] "RemoveContainer" containerID="c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa"
Sep 18 20:46:35.748671 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.748655    2583 scope.go:117] "RemoveContainer" containerID="4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c"
Sep 18 20:46:35.756070 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.756033    2583 scope.go:117] "RemoveContainer" containerID="468274e6a51f888ffb509d897603f0a9bcf8e47772a79cf0ead8a206a8e51708"
Sep 18 20:46:35.758323 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.758304    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"]
Sep 18 20:46:35.763609 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.763590    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gz77f"]
Sep 18 20:46:35.764350 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.764333    2583 scope.go:117] "RemoveContainer" containerID="c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa"
Sep 18 20:46:35.764638 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:46:35.764606    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa\": container with ID starting with c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa not found: ID does not exist" containerID="c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa"
Sep 18 20:46:35.764707 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.764650    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa"} err="failed to get container status \"c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa\": rpc error: code = NotFound desc = could not find container \"c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa\": container with ID starting with c201c2c23accd31c67163598b083936293b0df4c118f3b03864aae4620b758fa not found: ID does not exist"
Sep 18 20:46:35.764707 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.764677    2583 scope.go:117] "RemoveContainer" containerID="4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c"
Sep 18 20:46:35.764979 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:46:35.764962    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c\": container with ID starting with 4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c not found: ID does not exist" containerID="4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c"
Sep 18 20:46:35.765030 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.764986    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c"} err="failed to get container status \"4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c\": rpc error: code = NotFound desc = could not find container \"4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c\": container with ID starting with 4bb1d82b7e461a286d6a347f8a264cde288ae8143b58a9c000789d6df4bf9a5c not found: ID does not exist"
Sep 18 20:46:35.765030 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.765002    2583 scope.go:117] "RemoveContainer" containerID="468274e6a51f888ffb509d897603f0a9bcf8e47772a79cf0ead8a206a8e51708"
Sep 18 20:46:35.765280 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:46:35.765262    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"468274e6a51f888ffb509d897603f0a9bcf8e47772a79cf0ead8a206a8e51708\": container with ID starting with 468274e6a51f888ffb509d897603f0a9bcf8e47772a79cf0ead8a206a8e51708 not found: ID does not exist" containerID="468274e6a51f888ffb509d897603f0a9bcf8e47772a79cf0ead8a206a8e51708"
Sep 18 20:46:35.765321 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:35.765286    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468274e6a51f888ffb509d897603f0a9bcf8e47772a79cf0ead8a206a8e51708"} err="failed to get container status \"468274e6a51f888ffb509d897603f0a9bcf8e47772a79cf0ead8a206a8e51708\": rpc error: code = NotFound desc = could not find container \"468274e6a51f888ffb509d897603f0a9bcf8e47772a79cf0ead8a206a8e51708\": container with ID starting with 468274e6a51f888ffb509d897603f0a9bcf8e47772a79cf0ead8a206a8e51708 not found: ID does not exist"
Sep 18 20:46:37.315697 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:46:37.315665    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" path="/var/lib/kubelet/pods/cbcdcafb-2285-4936-aa51-4a84c387f65d/volumes"
Sep 18 20:47:07.273468 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.273431    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"]
Sep 18 20:47:07.273907 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.273796    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="kube-rbac-proxy"
Sep 18 20:47:07.273907 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.273807    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="kube-rbac-proxy"
Sep 18 20:47:07.273907 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.273821    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="storage-initializer"
Sep 18 20:47:07.273907 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.273826    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="storage-initializer"
Sep 18 20:47:07.273907 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.273832    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="kserve-container"
Sep 18 20:47:07.273907 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.273837    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="kserve-container"
Sep 18 20:47:07.273907 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.273893    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="kube-rbac-proxy"
Sep 18 20:47:07.273907 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.273904    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbcdcafb-2285-4936-aa51-4a84c387f65d" containerName="kserve-container"
Sep 18 20:47:07.276669 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.276650    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.279226 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.279193    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 18 20:47:07.279348 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.279258    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:47:07.279490 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.279476    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:47:07.279719 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.279700    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:47:07.280392 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.280374    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:47:07.280493 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.280381    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:47:07.280493 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.280408    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:47:07.287199 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.287171    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"]
Sep 18 20:47:07.321441 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.321410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb626309-79e2-4121-937d-0eb9bec31dac-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.321578 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.321451    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb626309-79e2-4121-937d-0eb9bec31dac-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.321578 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.321485    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb626309-79e2-4121-937d-0eb9bec31dac-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.321578 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.321560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzmp\" (UniqueName: \"kubernetes.io/projected/bb626309-79e2-4121-937d-0eb9bec31dac-kube-api-access-gbzmp\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.321688 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.321609    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb626309-79e2-4121-937d-0eb9bec31dac-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.422406 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.422372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb626309-79e2-4121-937d-0eb9bec31dac-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.422406 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.422408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzmp\" (UniqueName: \"kubernetes.io/projected/bb626309-79e2-4121-937d-0eb9bec31dac-kube-api-access-gbzmp\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.422606 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.422442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb626309-79e2-4121-937d-0eb9bec31dac-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.422606 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.422504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb626309-79e2-4121-937d-0eb9bec31dac-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.422606 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.422547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb626309-79e2-4121-937d-0eb9bec31dac-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.422798 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:47:07.422605    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 18 20:47:07.422798 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:47:07.422692    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb626309-79e2-4121-937d-0eb9bec31dac-proxy-tls podName:bb626309-79e2-4121-937d-0eb9bec31dac nodeName:}" failed. No retries permitted until 2026-09-18 20:47:07.922666304 +0000 UTC m=+1578.830568563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bb626309-79e2-4121-937d-0eb9bec31dac-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" (UID: "bb626309-79e2-4121-937d-0eb9bec31dac") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 18 20:47:07.422798 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.422783    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb626309-79e2-4121-937d-0eb9bec31dac-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.423315 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.423293    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb626309-79e2-4121-937d-0eb9bec31dac-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.423364 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.423330    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb626309-79e2-4121-937d-0eb9bec31dac-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.431439 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.431418    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzmp\" (UniqueName: \"kubernetes.io/projected/bb626309-79e2-4121-937d-0eb9bec31dac-kube-api-access-gbzmp\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.928401 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.928366    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb626309-79e2-4121-937d-0eb9bec31dac-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:07.930901 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:07.930878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb626309-79e2-4121-937d-0eb9bec31dac-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:08.188838 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:08.188753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:08.459832 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:08.459729    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"]
Sep 18 20:47:08.462667 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:47:08.462637    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb626309_79e2_4121_937d_0eb9bec31dac.slice/crio-4be5d61e19625de15573401daf346332d8547c09ab2061a39a9c6e153cf9b199 WatchSource:0}: Error finding container 4be5d61e19625de15573401daf346332d8547c09ab2061a39a9c6e153cf9b199: Status 404 returned error can't find the container with id 4be5d61e19625de15573401daf346332d8547c09ab2061a39a9c6e153cf9b199
Sep 18 20:47:08.891248 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:08.891117    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" event={"ID":"bb626309-79e2-4121-937d-0eb9bec31dac","Type":"ContainerStarted","Data":"f87458a203154126c5d60e48c46f5287dbae88e7d1d4fdf2f4ed294a875ef384"}
Sep 18 20:47:08.891248 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:08.891212    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" event={"ID":"bb626309-79e2-4121-937d-0eb9bec31dac","Type":"ContainerStarted","Data":"4be5d61e19625de15573401daf346332d8547c09ab2061a39a9c6e153cf9b199"}
Sep 18 20:47:13.914267 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:13.914229    2583 generic.go:356] "Generic (PLEG): container finished" podID="bb626309-79e2-4121-937d-0eb9bec31dac" containerID="f87458a203154126c5d60e48c46f5287dbae88e7d1d4fdf2f4ed294a875ef384" exitCode=0
Sep 18 20:47:13.914651 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:13.914304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" event={"ID":"bb626309-79e2-4121-937d-0eb9bec31dac","Type":"ContainerDied","Data":"f87458a203154126c5d60e48c46f5287dbae88e7d1d4fdf2f4ed294a875ef384"}
Sep 18 20:47:14.921390 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:14.921357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" event={"ID":"bb626309-79e2-4121-937d-0eb9bec31dac","Type":"ContainerStarted","Data":"e54b65ebf21e92c80a58971448280f233e10211db05352eb9ecb4d2bc80d5b28"}
Sep 18 20:47:14.921860 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:14.921425    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" event={"ID":"bb626309-79e2-4121-937d-0eb9bec31dac","Type":"ContainerStarted","Data":"9dc566954920babbcf6c8e8e01362a73010d28e35ae2016372e0883365fd1f85"}
Sep 18 20:47:14.921860 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:14.921648    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:14.942178 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:14.942136    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" podStartSLOduration=3.43537023 podStartE2EDuration="7.942106019s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.506735789s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:47:14.941062526 +0000 UTC m=+1585.848964844" watchObservedRunningTime="2026-09-18 20:47:14.942106019 +0000 UTC m=+1585.850008298"
Sep 18 20:47:15.926729 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:15.926699    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:21.936985 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:21.936954    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:47:52.024814 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:47:52.024725    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 18 20:48:01.940329 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:01.940300    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:48:07.329104 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:07.329070    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"]
Sep 18 20:48:07.329616 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:07.329334    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="kserve-container" containerID="cri-o://9dc566954920babbcf6c8e8e01362a73010d28e35ae2016372e0883365fd1f85" gracePeriod=30
Sep 18 20:48:07.329616 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:07.329374    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="kube-rbac-proxy" containerID="cri-o://e54b65ebf21e92c80a58971448280f233e10211db05352eb9ecb4d2bc80d5b28" gracePeriod=30
Sep 18 20:48:08.172878 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:08.172844    2583 generic.go:356] "Generic (PLEG): container finished" podID="bb626309-79e2-4121-937d-0eb9bec31dac" containerID="e54b65ebf21e92c80a58971448280f233e10211db05352eb9ecb4d2bc80d5b28" exitCode=2
Sep 18 20:48:08.172878 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:08.172886    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" event={"ID":"bb626309-79e2-4121-937d-0eb9bec31dac","Type":"ContainerDied","Data":"e54b65ebf21e92c80a58971448280f233e10211db05352eb9ecb4d2bc80d5b28"}
Sep 18 20:48:11.931806 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:11.931759    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 18 20:48:11.937611 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:11.937582    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 20:48:14.204237 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.204204    2583 generic.go:356] "Generic (PLEG): container finished" podID="bb626309-79e2-4121-937d-0eb9bec31dac" containerID="9dc566954920babbcf6c8e8e01362a73010d28e35ae2016372e0883365fd1f85" exitCode=0
Sep 18 20:48:14.204597 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.204275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" event={"ID":"bb626309-79e2-4121-937d-0eb9bec31dac","Type":"ContainerDied","Data":"9dc566954920babbcf6c8e8e01362a73010d28e35ae2016372e0883365fd1f85"}
Sep 18 20:48:14.330063 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.330043    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:48:14.500411 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.500387    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb626309-79e2-4121-937d-0eb9bec31dac-kserve-provision-location\") pod \"bb626309-79e2-4121-937d-0eb9bec31dac\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") "
Sep 18 20:48:14.500577 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.500441    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbzmp\" (UniqueName: \"kubernetes.io/projected/bb626309-79e2-4121-937d-0eb9bec31dac-kube-api-access-gbzmp\") pod \"bb626309-79e2-4121-937d-0eb9bec31dac\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") "
Sep 18 20:48:14.500577 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.500481    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb626309-79e2-4121-937d-0eb9bec31dac-cabundle-cert\") pod \"bb626309-79e2-4121-937d-0eb9bec31dac\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") "
Sep 18 20:48:14.500577 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.500504    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb626309-79e2-4121-937d-0eb9bec31dac-proxy-tls\") pod \"bb626309-79e2-4121-937d-0eb9bec31dac\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") "
Sep 18 20:48:14.500577 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.500529    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb626309-79e2-4121-937d-0eb9bec31dac-kube-rbac-proxy-sar-config\") pod \"bb626309-79e2-4121-937d-0eb9bec31dac\" (UID: \"bb626309-79e2-4121-937d-0eb9bec31dac\") "
Sep 18 20:48:14.500744 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.500714    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb626309-79e2-4121-937d-0eb9bec31dac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb626309-79e2-4121-937d-0eb9bec31dac" (UID: "bb626309-79e2-4121-937d-0eb9bec31dac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:48:14.500834 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.500809    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb626309-79e2-4121-937d-0eb9bec31dac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bb626309-79e2-4121-937d-0eb9bec31dac" (UID: "bb626309-79e2-4121-937d-0eb9bec31dac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:48:14.501037 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.501017    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb626309-79e2-4121-937d-0eb9bec31dac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bb626309-79e2-4121-937d-0eb9bec31dac" (UID: "bb626309-79e2-4121-937d-0eb9bec31dac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:48:14.502731 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.502700    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb626309-79e2-4121-937d-0eb9bec31dac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb626309-79e2-4121-937d-0eb9bec31dac" (UID: "bb626309-79e2-4121-937d-0eb9bec31dac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:48:14.502808 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.502722    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb626309-79e2-4121-937d-0eb9bec31dac-kube-api-access-gbzmp" (OuterVolumeSpecName: "kube-api-access-gbzmp") pod "bb626309-79e2-4121-937d-0eb9bec31dac" (UID: "bb626309-79e2-4121-937d-0eb9bec31dac"). InnerVolumeSpecName "kube-api-access-gbzmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:48:14.602000 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.601967    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb626309-79e2-4121-937d-0eb9bec31dac-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:48:14.602000 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.601994    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzmp\" (UniqueName: \"kubernetes.io/projected/bb626309-79e2-4121-937d-0eb9bec31dac-kube-api-access-gbzmp\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:48:14.602000 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.602005    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb626309-79e2-4121-937d-0eb9bec31dac-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:48:14.602250 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.602014    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb626309-79e2-4121-937d-0eb9bec31dac-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:48:14.602250 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:14.602021    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb626309-79e2-4121-937d-0eb9bec31dac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:48:15.211341 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:15.211302    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5" event={"ID":"bb626309-79e2-4121-937d-0eb9bec31dac","Type":"ContainerDied","Data":"4be5d61e19625de15573401daf346332d8547c09ab2061a39a9c6e153cf9b199"}
Sep 18 20:48:15.211747 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:15.211346    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"
Sep 18 20:48:15.211747 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:15.211382    2583 scope.go:117] "RemoveContainer" containerID="e54b65ebf21e92c80a58971448280f233e10211db05352eb9ecb4d2bc80d5b28"
Sep 18 20:48:15.220519 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:15.220503    2583 scope.go:117] "RemoveContainer" containerID="9dc566954920babbcf6c8e8e01362a73010d28e35ae2016372e0883365fd1f85"
Sep 18 20:48:15.228095 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:15.228080    2583 scope.go:117] "RemoveContainer" containerID="f87458a203154126c5d60e48c46f5287dbae88e7d1d4fdf2f4ed294a875ef384"
Sep 18 20:48:15.233047 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:15.233023    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"]
Sep 18 20:48:15.237162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:15.237142    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2hrf5"]
Sep 18 20:48:15.316020 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:15.315991    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" path="/var/lib/kubelet/pods/bb626309-79e2-4121-937d-0eb9bec31dac/volumes"
Sep 18 20:48:37.882963 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.882930    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"]
Sep 18 20:48:37.883470 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.883357    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="kube-rbac-proxy"
Sep 18 20:48:37.883470 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.883376    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="kube-rbac-proxy"
Sep 18 20:48:37.883470 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.883404    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="storage-initializer"
Sep 18 20:48:37.883470 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.883410    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="storage-initializer"
Sep 18 20:48:37.883470 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.883428    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="kserve-container"
Sep 18 20:48:37.883470 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.883437    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="kserve-container"
Sep 18 20:48:37.883757 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.883522    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="kube-rbac-proxy"
Sep 18 20:48:37.883757 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.883536    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb626309-79e2-4121-937d-0eb9bec31dac" containerName="kserve-container"
Sep 18 20:48:37.887605 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.887589    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:37.890381 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.890357    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:48:37.890512 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.890487    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:48:37.890585 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.890573    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 20:48:37.890712 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.890692    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 20:48:37.890781 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.890762    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:48:37.890934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.890916    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:48:37.891040 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.890917    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:48:37.896886 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.896864    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"]
Sep 18 20:48:37.994930 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.994862    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:37.995162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.994993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5xsk\" (UniqueName: \"kubernetes.io/projected/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kube-api-access-g5xsk\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:37.995162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.995030    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:37.995162 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.995093    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:37.995328 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:37.995161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:38.096447 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:38.096404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:38.096662 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:38.096458    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:38.096662 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:38.096649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:38.096805 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:38.096722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:38.096805 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:38.096755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5xsk\" (UniqueName: \"kubernetes.io/projected/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kube-api-access-g5xsk\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:38.096963 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:38.096937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:38.097309 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:38.097292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:38.097391 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:38.097374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:38.099177 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:38.099155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:38.105748 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:38.105728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5xsk\" (UniqueName: \"kubernetes.io/projected/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kube-api-access-g5xsk\") pod \"isvc-canary-create-predictor-76b74f4d6c-x69jc\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:38.198002 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:38.197970    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:38.532722 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:38.532698    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"]
Sep 18 20:48:38.534807 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:48:38.534780    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc158d9f2_798c_4af9_a0b3_5a92a18cf38f.slice/crio-6d08fe31a5bb8afbc37cf32f167173ef1a4894729a67ce43ef3348a586f8b216 WatchSource:0}: Error finding container 6d08fe31a5bb8afbc37cf32f167173ef1a4894729a67ce43ef3348a586f8b216: Status 404 returned error can't find the container with id 6d08fe31a5bb8afbc37cf32f167173ef1a4894729a67ce43ef3348a586f8b216
Sep 18 20:48:39.320585 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:39.320543    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" event={"ID":"c158d9f2-798c-4af9-a0b3-5a92a18cf38f","Type":"ContainerStarted","Data":"35d98efedec10f636b833ad4580919a848c52ca327bf3b7185de62fb3ac9fc8d"}
Sep 18 20:48:39.321059 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:39.320603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" event={"ID":"c158d9f2-798c-4af9-a0b3-5a92a18cf38f","Type":"ContainerStarted","Data":"6d08fe31a5bb8afbc37cf32f167173ef1a4894729a67ce43ef3348a586f8b216"}
Sep 18 20:48:43.340412 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:43.340374    2583 generic.go:356] "Generic (PLEG): container finished" podID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerID="35d98efedec10f636b833ad4580919a848c52ca327bf3b7185de62fb3ac9fc8d" exitCode=0
Sep 18 20:48:43.340814 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:43.340440    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" event={"ID":"c158d9f2-798c-4af9-a0b3-5a92a18cf38f","Type":"ContainerDied","Data":"35d98efedec10f636b833ad4580919a848c52ca327bf3b7185de62fb3ac9fc8d"}
Sep 18 20:48:43.341636 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:43.341622    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:48:57.446510 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:57.446474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" event={"ID":"c158d9f2-798c-4af9-a0b3-5a92a18cf38f","Type":"ContainerStarted","Data":"4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab"}
Sep 18 20:48:57.446983 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:57.446539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" event={"ID":"c158d9f2-798c-4af9-a0b3-5a92a18cf38f","Type":"ContainerStarted","Data":"80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30"}
Sep 18 20:48:57.446983 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:57.446750    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:57.446983 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:57.446897    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:48:57.448485 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:57.448456    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:48:57.465849 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:57.465799    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podStartSLOduration=3.214639636 podStartE2EDuration="20.465786292s" totalImagesPullingTime="13.296911863s" totalInitContainerRuntime="3.954234793s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:48:57.463833837 +0000 UTC m=+1688.371736115" watchObservedRunningTime="2026-09-18 20:48:57.465786292 +0000 UTC m=+1688.373688639"
Sep 18 20:48:58.450713 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:48:58.450670    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:49:03.455444 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:49:03.455372    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:49:03.455943 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:49:03.455914    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:49:13.456311 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:49:13.456270    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:49:23.455959 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:49:23.455919    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:49:33.456028 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:49:33.455985    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:49:43.456457 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:49:43.456419    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:49:53.456053 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:49:53.456007    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:50:03.456039 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:03.455994    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:50:06.311142 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:06.311086    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:50:16.312874 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:16.312845    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:50:17.859027 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:17.858995    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"]
Sep 18 20:50:17.859428 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:17.859255    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container" containerID="cri-o://80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30" gracePeriod=30
Sep 18 20:50:17.859428 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:17.859303    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kube-rbac-proxy" containerID="cri-o://4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab" gracePeriod=30
Sep 18 20:50:18.451115 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:18.451068    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 18 20:50:18.815200 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:18.815083    2583 generic.go:356] "Generic (PLEG): container finished" podID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerID="4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab" exitCode=2
Sep 18 20:50:18.815200 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:18.815159    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" event={"ID":"c158d9f2-798c-4af9-a0b3-5a92a18cf38f","Type":"ContainerDied","Data":"4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab"}
Sep 18 20:50:22.605390 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.605368    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:50:22.725513 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.725413    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-proxy-tls\") pod \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") "
Sep 18 20:50:22.725513 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.725471    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5xsk\" (UniqueName: \"kubernetes.io/projected/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kube-api-access-g5xsk\") pod \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") "
Sep 18 20:50:22.725513 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.725499    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-cabundle-cert\") pod \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") "
Sep 18 20:50:22.725799 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.725536    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kube-rbac-proxy-sar-config\") pod \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") "
Sep 18 20:50:22.725799 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.725559    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kserve-provision-location\") pod \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\" (UID: \"c158d9f2-798c-4af9-a0b3-5a92a18cf38f\") "
Sep 18 20:50:22.725938 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.725917    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c158d9f2-798c-4af9-a0b3-5a92a18cf38f" (UID: "c158d9f2-798c-4af9-a0b3-5a92a18cf38f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:50:22.725938 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.725924    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c158d9f2-798c-4af9-a0b3-5a92a18cf38f" (UID: "c158d9f2-798c-4af9-a0b3-5a92a18cf38f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:50:22.726032 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.725936    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c158d9f2-798c-4af9-a0b3-5a92a18cf38f" (UID: "c158d9f2-798c-4af9-a0b3-5a92a18cf38f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:50:22.727669 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.727649    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c158d9f2-798c-4af9-a0b3-5a92a18cf38f" (UID: "c158d9f2-798c-4af9-a0b3-5a92a18cf38f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:50:22.727787 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.727771    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kube-api-access-g5xsk" (OuterVolumeSpecName: "kube-api-access-g5xsk") pod "c158d9f2-798c-4af9-a0b3-5a92a18cf38f" (UID: "c158d9f2-798c-4af9-a0b3-5a92a18cf38f"). InnerVolumeSpecName "kube-api-access-g5xsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:50:22.826501 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.826469    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:50:22.826501 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.826494    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:50:22.826501 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.826505    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:50:22.826714 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.826514    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:50:22.826714 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.826523    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5xsk\" (UniqueName: \"kubernetes.io/projected/c158d9f2-798c-4af9-a0b3-5a92a18cf38f-kube-api-access-g5xsk\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:50:22.835734 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.835706    2583 generic.go:356] "Generic (PLEG): container finished" podID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerID="80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30" exitCode=0
Sep 18 20:50:22.835894 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.835778    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"
Sep 18 20:50:22.835894 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.835799    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" event={"ID":"c158d9f2-798c-4af9-a0b3-5a92a18cf38f","Type":"ContainerDied","Data":"80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30"}
Sep 18 20:50:22.835894 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.835859    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc" event={"ID":"c158d9f2-798c-4af9-a0b3-5a92a18cf38f","Type":"ContainerDied","Data":"6d08fe31a5bb8afbc37cf32f167173ef1a4894729a67ce43ef3348a586f8b216"}
Sep 18 20:50:22.835894 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.835885    2583 scope.go:117] "RemoveContainer" containerID="4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab"
Sep 18 20:50:22.844769 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.844753    2583 scope.go:117] "RemoveContainer" containerID="80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30"
Sep 18 20:50:22.852262 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.852241    2583 scope.go:117] "RemoveContainer" containerID="35d98efedec10f636b833ad4580919a848c52ca327bf3b7185de62fb3ac9fc8d"
Sep 18 20:50:22.858225 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.858203    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"]
Sep 18 20:50:22.860788 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.860770    2583 scope.go:117] "RemoveContainer" containerID="4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab"
Sep 18 20:50:22.861060 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:50:22.861045    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab\": container with ID starting with 4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab not found: ID does not exist" containerID="4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab"
Sep 18 20:50:22.861140 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.861067    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab"} err="failed to get container status \"4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab\": rpc error: code = NotFound desc = could not find container \"4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab\": container with ID starting with 4ea679b04ec5ee0cd926d0652779345038dc7f6677b01cc71c8ac3bed6719aab not found: ID does not exist"
Sep 18 20:50:22.861140 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.861084    2583 scope.go:117] "RemoveContainer" containerID="80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30"
Sep 18 20:50:22.861383 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:50:22.861357    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30\": container with ID starting with 80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30 not found: ID does not exist" containerID="80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30"
Sep 18 20:50:22.861383 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.861390    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30"} err="failed to get container status \"80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30\": rpc error: code = NotFound desc = could not find container \"80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30\": container with ID starting with 80ec42828e42ab0d28168de737249242f69af8923515b5a00a7aeda78d5e5a30 not found: ID does not exist"
Sep 18 20:50:22.861383 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.861405    2583 scope.go:117] "RemoveContainer" containerID="35d98efedec10f636b833ad4580919a848c52ca327bf3b7185de62fb3ac9fc8d"
Sep 18 20:50:22.861651 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:50:22.861629    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d98efedec10f636b833ad4580919a848c52ca327bf3b7185de62fb3ac9fc8d\": container with ID starting with 35d98efedec10f636b833ad4580919a848c52ca327bf3b7185de62fb3ac9fc8d not found: ID does not exist" containerID="35d98efedec10f636b833ad4580919a848c52ca327bf3b7185de62fb3ac9fc8d"
Sep 18 20:50:22.861728 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.861654    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d98efedec10f636b833ad4580919a848c52ca327bf3b7185de62fb3ac9fc8d"} err="failed to get container status \"35d98efedec10f636b833ad4580919a848c52ca327bf3b7185de62fb3ac9fc8d\": rpc error: code = NotFound desc = could not find container \"35d98efedec10f636b833ad4580919a848c52ca327bf3b7185de62fb3ac9fc8d\": container with ID starting with 35d98efedec10f636b833ad4580919a848c52ca327bf3b7185de62fb3ac9fc8d not found: ID does not exist"
Sep 18 20:50:22.862237 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:22.862220    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-x69jc"]
Sep 18 20:50:23.044516 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.044435    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"]
Sep 18 20:50:23.044917 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.044896    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container"
Sep 18 20:50:23.044917 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.044916    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container"
Sep 18 20:50:23.045086 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.044936    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="storage-initializer"
Sep 18 20:50:23.045086 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.044941    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="storage-initializer"
Sep 18 20:50:23.045086 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.044948    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kube-rbac-proxy"
Sep 18 20:50:23.045086 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.044953    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kube-rbac-proxy"
Sep 18 20:50:23.045086 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.045030    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kube-rbac-proxy"
Sep 18 20:50:23.045086 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.045042    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" containerName="kserve-container"
Sep 18 20:50:23.049559 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.049539    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.052470 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.052449    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:50:23.052632 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.052556    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 20:50:23.052632 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.052590    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:50:23.052632 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.052567    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:50:23.052812 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.052668    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:50:23.052930 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.052914    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:50:23.053000 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.052927    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 20:50:23.058979 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.058960    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"]
Sep 18 20:50:23.128420 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.128386    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"]
Sep 18 20:50:23.129831 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.129805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.129961 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.129844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5gks\" (UniqueName: \"kubernetes.io/projected/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kube-api-access-g5gks\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.129961 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.129878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.130047 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.129977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.130047 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.130032    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.132384 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.132370    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.135179 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.135160    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 20:50:23.142712 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.142691    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"]
Sep 18 20:50:23.230539 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.230503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6236e506-8c80-435d-8a0a-e0893d1ff712-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.230539 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.230543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wnsq\" (UniqueName: \"kubernetes.io/projected/6236e506-8c80-435d-8a0a-e0893d1ff712-kube-api-access-4wnsq\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.230787 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.230595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.230787 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.230640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6236e506-8c80-435d-8a0a-e0893d1ff712-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.230787 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.230701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.230787 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.230737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5gks\" (UniqueName: \"kubernetes.io/projected/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kube-api-access-g5gks\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.230787 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.230770    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.231048 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.230807    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6236e506-8c80-435d-8a0a-e0893d1ff712-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.231048 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.230856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6236e506-8c80-435d-8a0a-e0893d1ff712-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.231048 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.230897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.231318 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.231297    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.231784 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.231758    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.231860 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.231758    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.233934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.233910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.238841 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.238820    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5gks\" (UniqueName: \"kubernetes.io/projected/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kube-api-access-g5gks\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.317149 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.317063    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c158d9f2-798c-4af9-a0b3-5a92a18cf38f" path="/var/lib/kubelet/pods/c158d9f2-798c-4af9-a0b3-5a92a18cf38f/volumes"
Sep 18 20:50:23.332292 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.332259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6236e506-8c80-435d-8a0a-e0893d1ff712-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.332404 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.332339    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6236e506-8c80-435d-8a0a-e0893d1ff712-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.332404 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.332364    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6236e506-8c80-435d-8a0a-e0893d1ff712-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.332404 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.332399    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6236e506-8c80-435d-8a0a-e0893d1ff712-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.332506 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.332424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wnsq\" (UniqueName: \"kubernetes.io/projected/6236e506-8c80-435d-8a0a-e0893d1ff712-kube-api-access-4wnsq\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.332825 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.332798    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6236e506-8c80-435d-8a0a-e0893d1ff712-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.332936 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.332917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6236e506-8c80-435d-8a0a-e0893d1ff712-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.333050 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.333028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6236e506-8c80-435d-8a0a-e0893d1ff712-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.334916 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.334897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6236e506-8c80-435d-8a0a-e0893d1ff712-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.340594 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.340574    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wnsq\" (UniqueName: \"kubernetes.io/projected/6236e506-8c80-435d-8a0a-e0893d1ff712-kube-api-access-4wnsq\") pod \"isvc-canary-promote-predictor-6546f8697f-722qz\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.360952 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.360931    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:23.443175 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.443146    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:23.632669 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.632642    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"]
Sep 18 20:50:23.634840 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:50:23.634813    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8302771_4bc6_4e38_b6a0_eff6e523f3c4.slice/crio-a95bc1eb536844871b9661e765019dc7bc66563e812c29dec44f8c337a640345 WatchSource:0}: Error finding container a95bc1eb536844871b9661e765019dc7bc66563e812c29dec44f8c337a640345: Status 404 returned error can't find the container with id a95bc1eb536844871b9661e765019dc7bc66563e812c29dec44f8c337a640345
Sep 18 20:50:23.712576 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.712554    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"]
Sep 18 20:50:23.714874 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:50:23.714852    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6236e506_8c80_435d_8a0a_e0893d1ff712.slice/crio-d46a40f4cda45e68ef3a257e09139c932b6e9c28107743c1ee14a0b04190fb31 WatchSource:0}: Error finding container d46a40f4cda45e68ef3a257e09139c932b6e9c28107743c1ee14a0b04190fb31: Status 404 returned error can't find the container with id d46a40f4cda45e68ef3a257e09139c932b6e9c28107743c1ee14a0b04190fb31
Sep 18 20:50:23.843087 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.843049    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" event={"ID":"a8302771-4bc6-4e38-b6a0-eff6e523f3c4","Type":"ContainerStarted","Data":"826c53c32e50e32064badbc4a6dbb9a3a499713a034b4d60aca9e54e30f1dc51"}
Sep 18 20:50:23.843298 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.843151    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" event={"ID":"a8302771-4bc6-4e38-b6a0-eff6e523f3c4","Type":"ContainerStarted","Data":"a95bc1eb536844871b9661e765019dc7bc66563e812c29dec44f8c337a640345"}
Sep 18 20:50:23.845846 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.845813    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" event={"ID":"6236e506-8c80-435d-8a0a-e0893d1ff712","Type":"ContainerStarted","Data":"a3d403c79889cfd779ecd2e72b09cdec3a78b61fa9c046ba70b8726601071da3"}
Sep 18 20:50:23.845965 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:23.845863    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" event={"ID":"6236e506-8c80-435d-8a0a-e0893d1ff712","Type":"ContainerStarted","Data":"d46a40f4cda45e68ef3a257e09139c932b6e9c28107743c1ee14a0b04190fb31"}
Sep 18 20:50:27.865263 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:27.865232    2583 generic.go:356] "Generic (PLEG): container finished" podID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerID="a3d403c79889cfd779ecd2e72b09cdec3a78b61fa9c046ba70b8726601071da3" exitCode=0
Sep 18 20:50:27.865801 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:27.865300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" event={"ID":"6236e506-8c80-435d-8a0a-e0893d1ff712","Type":"ContainerDied","Data":"a3d403c79889cfd779ecd2e72b09cdec3a78b61fa9c046ba70b8726601071da3"}
Sep 18 20:50:27.866973 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:27.866954    2583 generic.go:356] "Generic (PLEG): container finished" podID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerID="826c53c32e50e32064badbc4a6dbb9a3a499713a034b4d60aca9e54e30f1dc51" exitCode=0
Sep 18 20:50:27.867052 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:27.866997    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" event={"ID":"a8302771-4bc6-4e38-b6a0-eff6e523f3c4","Type":"ContainerDied","Data":"826c53c32e50e32064badbc4a6dbb9a3a499713a034b4d60aca9e54e30f1dc51"}
Sep 18 20:50:28.873896 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:28.873804    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" event={"ID":"6236e506-8c80-435d-8a0a-e0893d1ff712","Type":"ContainerStarted","Data":"161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e"}
Sep 18 20:50:28.874363 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:28.873917    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" event={"ID":"6236e506-8c80-435d-8a0a-e0893d1ff712","Type":"ContainerStarted","Data":"7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c"}
Sep 18 20:50:28.874363 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:28.874159    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:28.876033 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:28.876009    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" event={"ID":"a8302771-4bc6-4e38-b6a0-eff6e523f3c4","Type":"ContainerStarted","Data":"c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573"}
Sep 18 20:50:28.876181 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:28.876064    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" event={"ID":"a8302771-4bc6-4e38-b6a0-eff6e523f3c4","Type":"ContainerStarted","Data":"c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c"}
Sep 18 20:50:28.876292 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:28.876274    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:28.893390 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:28.893348    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podStartSLOduration=1.988753772 podStartE2EDuration="5.89333613s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.904582358s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:50:28.891543241 +0000 UTC m=+1779.799445524" watchObservedRunningTime="2026-09-18 20:50:28.89333613 +0000 UTC m=+1779.801238410"
Sep 18 20:50:28.910768 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:28.910725    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podStartSLOduration=2.243073079 podStartE2EDuration="5.910712713s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.667639634s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:50:28.909035665 +0000 UTC m=+1779.816937946" watchObservedRunningTime="2026-09-18 20:50:28.910712713 +0000 UTC m=+1779.818614992"
Sep 18 20:50:29.882195 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:29.882154    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:29.882650 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:29.882223    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:29.883276 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:29.883245    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:50:29.883391 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:29.883287    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:50:30.886947 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:30.886907    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:50:30.887452 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:30.887037    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:50:35.891920 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:35.891890    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:50:35.892444 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:35.892195    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:50:35.892526 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:35.892501    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:50:35.892839 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:35.892815    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:50:45.893406 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:45.893361    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:50:45.893406 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:45.893396    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:50:51.372070 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:51.372043    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:50:51.374014 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:51.373989    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:50:55.892971 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:55.892931    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:50:55.893466 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:50:55.892931    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:51:05.892505 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:05.892465    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:51:05.893025 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:05.892781    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:51:15.893187 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:15.893148    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:51:15.893735 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:15.893147    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:51:24.485208 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.485173    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"]
Sep 18 20:51:24.488897 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.488879    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.491781 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.491755    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 18 20:51:24.491887 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.491785    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 18 20:51:24.498234 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.498211    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"]
Sep 18 20:51:24.579264 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.579231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c923f5-2a76-4e89-a75e-7427dc033d16-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.579440 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.579338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35c923f5-2a76-4e89-a75e-7427dc033d16-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.579440 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.579383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c923f5-2a76-4e89-a75e-7427dc033d16-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.579440 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.579399    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p82fm\" (UniqueName: \"kubernetes.io/projected/35c923f5-2a76-4e89-a75e-7427dc033d16-kube-api-access-p82fm\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.579440 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.579427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c923f5-2a76-4e89-a75e-7427dc033d16-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.680066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.680034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c923f5-2a76-4e89-a75e-7427dc033d16-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.680230 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.680108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35c923f5-2a76-4e89-a75e-7427dc033d16-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.680230 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.680163    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c923f5-2a76-4e89-a75e-7427dc033d16-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.680230 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.680178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82fm\" (UniqueName: \"kubernetes.io/projected/35c923f5-2a76-4e89-a75e-7427dc033d16-kube-api-access-p82fm\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.680230 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.680218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c923f5-2a76-4e89-a75e-7427dc033d16-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.680415 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:51:24.680265    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 18 20:51:24.680415 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:51:24.680334    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35c923f5-2a76-4e89-a75e-7427dc033d16-proxy-tls podName:35c923f5-2a76-4e89-a75e-7427dc033d16 nodeName:}" failed. No retries permitted until 2026-09-18 20:51:25.18031336 +0000 UTC m=+1836.088215654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/35c923f5-2a76-4e89-a75e-7427dc033d16-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-mj6mx" (UID: "35c923f5-2a76-4e89-a75e-7427dc033d16") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 18 20:51:24.680527 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.680508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c923f5-2a76-4e89-a75e-7427dc033d16-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.680808 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.680792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35c923f5-2a76-4e89-a75e-7427dc033d16-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.680854 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.680841    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c923f5-2a76-4e89-a75e-7427dc033d16-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:24.689663 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:24.689643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p82fm\" (UniqueName: \"kubernetes.io/projected/35c923f5-2a76-4e89-a75e-7427dc033d16-kube-api-access-p82fm\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:25.184397 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:25.184364    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c923f5-2a76-4e89-a75e-7427dc033d16-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:25.186879 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:25.186855    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c923f5-2a76-4e89-a75e-7427dc033d16-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-mj6mx\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:25.400929 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:25.400880    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:25.677066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:25.677036    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"]
Sep 18 20:51:25.679472 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:51:25.679443    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c923f5_2a76_4e89_a75e_7427dc033d16.slice/crio-6aee69bfd7879cafefa59e320ed4b8af957a5835b6d52565ec93a0de78aa79d7 WatchSource:0}: Error finding container 6aee69bfd7879cafefa59e320ed4b8af957a5835b6d52565ec93a0de78aa79d7: Status 404 returned error can't find the container with id 6aee69bfd7879cafefa59e320ed4b8af957a5835b6d52565ec93a0de78aa79d7
Sep 18 20:51:25.892832 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:25.892792    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:51:25.893006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:25.892792    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:51:26.169668 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:26.169629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" event={"ID":"35c923f5-2a76-4e89-a75e-7427dc033d16","Type":"ContainerStarted","Data":"d96eff24c706a7205026b902d112ce40f208d6325073f8501c9f9907f064b441"}
Sep 18 20:51:26.169843 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:26.169695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" event={"ID":"35c923f5-2a76-4e89-a75e-7427dc033d16","Type":"ContainerStarted","Data":"6aee69bfd7879cafefa59e320ed4b8af957a5835b6d52565ec93a0de78aa79d7"}
Sep 18 20:51:30.189267 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:30.189237    2583 generic.go:356] "Generic (PLEG): container finished" podID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerID="d96eff24c706a7205026b902d112ce40f208d6325073f8501c9f9907f064b441" exitCode=0
Sep 18 20:51:30.189617 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:30.189301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" event={"ID":"35c923f5-2a76-4e89-a75e-7427dc033d16","Type":"ContainerDied","Data":"d96eff24c706a7205026b902d112ce40f208d6325073f8501c9f9907f064b441"}
Sep 18 20:51:34.216883 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:34.216798    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" event={"ID":"35c923f5-2a76-4e89-a75e-7427dc033d16","Type":"ContainerStarted","Data":"e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a"}
Sep 18 20:51:34.216883 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:34.216861    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" event={"ID":"35c923f5-2a76-4e89-a75e-7427dc033d16","Type":"ContainerStarted","Data":"99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651"}
Sep 18 20:51:34.217311 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:34.217142    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:34.217311 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:34.217187    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:34.218741 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:34.218717    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:51:34.237268 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:34.236741    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" podStartSLOduration=2.170737229 podStartE2EDuration="10.23672369s" totalImagesPullingTime="3.758751203s" totalInitContainerRuntime="4.307235258s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:51:34.234604907 +0000 UTC m=+1845.142507191" watchObservedRunningTime="2026-09-18 20:51:34.23672369 +0000 UTC m=+1845.144625969"
Sep 18 20:51:35.222595 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:35.222551    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:51:35.893117 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:35.893080    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:51:35.893117 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:35.893100    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:51:40.228301 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:40.228265    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:40.229283 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:40.229240    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:51:45.894180 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:45.894152    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:51:45.894692 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:45.894243    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:51:46.020262 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:46.018967    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"]
Sep 18 20:51:46.277921 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:46.277876    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container" containerID="cri-o://7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c" gracePeriod=30
Sep 18 20:51:46.277921 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:46.277903    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kube-rbac-proxy" containerID="cri-o://161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e" gracePeriod=30
Sep 18 20:51:47.284574 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:47.284541    2583 generic.go:356] "Generic (PLEG): container finished" podID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerID="161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e" exitCode=2
Sep 18 20:51:47.284953 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:47.284613    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" event={"ID":"6236e506-8c80-435d-8a0a-e0893d1ff712","Type":"ContainerDied","Data":"161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e"}
Sep 18 20:51:50.230116 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:50.230088    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:51:50.887916 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:50.887870    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 18 20:51:51.023923 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.023895    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:51:51.125361 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.125290    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6236e506-8c80-435d-8a0a-e0893d1ff712-proxy-tls\") pod \"6236e506-8c80-435d-8a0a-e0893d1ff712\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") "
Sep 18 20:51:51.125361 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.125348    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wnsq\" (UniqueName: \"kubernetes.io/projected/6236e506-8c80-435d-8a0a-e0893d1ff712-kube-api-access-4wnsq\") pod \"6236e506-8c80-435d-8a0a-e0893d1ff712\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") "
Sep 18 20:51:51.125582 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.125385    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6236e506-8c80-435d-8a0a-e0893d1ff712-cabundle-cert\") pod \"6236e506-8c80-435d-8a0a-e0893d1ff712\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") "
Sep 18 20:51:51.125582 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.125408    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6236e506-8c80-435d-8a0a-e0893d1ff712-kube-rbac-proxy-sar-config\") pod \"6236e506-8c80-435d-8a0a-e0893d1ff712\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") "
Sep 18 20:51:51.125582 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.125446    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6236e506-8c80-435d-8a0a-e0893d1ff712-kserve-provision-location\") pod \"6236e506-8c80-435d-8a0a-e0893d1ff712\" (UID: \"6236e506-8c80-435d-8a0a-e0893d1ff712\") "
Sep 18 20:51:51.125795 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.125769    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6236e506-8c80-435d-8a0a-e0893d1ff712-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6236e506-8c80-435d-8a0a-e0893d1ff712" (UID: "6236e506-8c80-435d-8a0a-e0893d1ff712"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:51:51.125851 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.125785    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6236e506-8c80-435d-8a0a-e0893d1ff712-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6236e506-8c80-435d-8a0a-e0893d1ff712" (UID: "6236e506-8c80-435d-8a0a-e0893d1ff712"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:51:51.125851 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.125812    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6236e506-8c80-435d-8a0a-e0893d1ff712-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6236e506-8c80-435d-8a0a-e0893d1ff712" (UID: "6236e506-8c80-435d-8a0a-e0893d1ff712"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:51.127625 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.127603    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6236e506-8c80-435d-8a0a-e0893d1ff712-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6236e506-8c80-435d-8a0a-e0893d1ff712" (UID: "6236e506-8c80-435d-8a0a-e0893d1ff712"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:51:51.127752 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.127730    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6236e506-8c80-435d-8a0a-e0893d1ff712-kube-api-access-4wnsq" (OuterVolumeSpecName: "kube-api-access-4wnsq") pod "6236e506-8c80-435d-8a0a-e0893d1ff712" (UID: "6236e506-8c80-435d-8a0a-e0893d1ff712"). InnerVolumeSpecName "kube-api-access-4wnsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:51:51.226584 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.226556    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wnsq\" (UniqueName: \"kubernetes.io/projected/6236e506-8c80-435d-8a0a-e0893d1ff712-kube-api-access-4wnsq\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:51:51.226584 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.226581    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6236e506-8c80-435d-8a0a-e0893d1ff712-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:51:51.226584 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.226591    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6236e506-8c80-435d-8a0a-e0893d1ff712-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:51:51.226799 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.226599    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6236e506-8c80-435d-8a0a-e0893d1ff712-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:51:51.226799 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.226608    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6236e506-8c80-435d-8a0a-e0893d1ff712-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:51:51.305138 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.305101    2583 generic.go:356] "Generic (PLEG): container finished" podID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerID="7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c" exitCode=0
Sep 18 20:51:51.305533 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.305184    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"
Sep 18 20:51:51.305533 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.305185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" event={"ID":"6236e506-8c80-435d-8a0a-e0893d1ff712","Type":"ContainerDied","Data":"7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c"}
Sep 18 20:51:51.305533 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.305326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz" event={"ID":"6236e506-8c80-435d-8a0a-e0893d1ff712","Type":"ContainerDied","Data":"d46a40f4cda45e68ef3a257e09139c932b6e9c28107743c1ee14a0b04190fb31"}
Sep 18 20:51:51.305533 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.305359    2583 scope.go:117] "RemoveContainer" containerID="161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e"
Sep 18 20:51:51.315639 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.315619    2583 scope.go:117] "RemoveContainer" containerID="7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c"
Sep 18 20:51:51.323696 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.323679    2583 scope.go:117] "RemoveContainer" containerID="a3d403c79889cfd779ecd2e72b09cdec3a78b61fa9c046ba70b8726601071da3"
Sep 18 20:51:51.331488 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.331469    2583 scope.go:117] "RemoveContainer" containerID="161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e"
Sep 18 20:51:51.331744 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:51:51.331726    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e\": container with ID starting with 161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e not found: ID does not exist" containerID="161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e"
Sep 18 20:51:51.331789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.331753    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e"} err="failed to get container status \"161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e\": rpc error: code = NotFound desc = could not find container \"161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e\": container with ID starting with 161fdf492b421a698890cfd02380ff4c2e6a0857f516543ba455a856da036c7e not found: ID does not exist"
Sep 18 20:51:51.331789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.331771    2583 scope.go:117] "RemoveContainer" containerID="7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c"
Sep 18 20:51:51.332004 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:51:51.331986    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c\": container with ID starting with 7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c not found: ID does not exist" containerID="7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c"
Sep 18 20:51:51.332050 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.332010    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c"} err="failed to get container status \"7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c\": rpc error: code = NotFound desc = could not find container \"7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c\": container with ID starting with 7eafd4a1d3814692bf90526fcdb7ce0a09fd98f43dd03ac2a2d88b2cbd74539c not found: ID does not exist"
Sep 18 20:51:51.332050 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.332025    2583 scope.go:117] "RemoveContainer" containerID="a3d403c79889cfd779ecd2e72b09cdec3a78b61fa9c046ba70b8726601071da3"
Sep 18 20:51:51.332313 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:51:51.332296    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d403c79889cfd779ecd2e72b09cdec3a78b61fa9c046ba70b8726601071da3\": container with ID starting with a3d403c79889cfd779ecd2e72b09cdec3a78b61fa9c046ba70b8726601071da3 not found: ID does not exist" containerID="a3d403c79889cfd779ecd2e72b09cdec3a78b61fa9c046ba70b8726601071da3"
Sep 18 20:51:51.332380 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.332318    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d403c79889cfd779ecd2e72b09cdec3a78b61fa9c046ba70b8726601071da3"} err="failed to get container status \"a3d403c79889cfd779ecd2e72b09cdec3a78b61fa9c046ba70b8726601071da3\": rpc error: code = NotFound desc = could not find container \"a3d403c79889cfd779ecd2e72b09cdec3a78b61fa9c046ba70b8726601071da3\": container with ID starting with a3d403c79889cfd779ecd2e72b09cdec3a78b61fa9c046ba70b8726601071da3 not found: ID does not exist"
Sep 18 20:51:51.336252 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.336224    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"]
Sep 18 20:51:51.343977 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:51.342199    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-722qz"]
Sep 18 20:51:53.316206 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:51:53.316173    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" path="/var/lib/kubelet/pods/6236e506-8c80-435d-8a0a-e0893d1ff712/volumes"
Sep 18 20:52:05.552051 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:05.551977    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"]
Sep 18 20:52:05.552479 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:05.552269    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kserve-container" containerID="cri-o://99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651" gracePeriod=30
Sep 18 20:52:05.552479 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:05.552298    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kube-rbac-proxy" containerID="cri-o://e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a" gracePeriod=30
Sep 18 20:52:06.381068 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:06.381035    2583 generic.go:356] "Generic (PLEG): container finished" podID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerID="e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a" exitCode=2
Sep 18 20:52:06.381273 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:06.381109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" event={"ID":"35c923f5-2a76-4e89-a75e-7427dc033d16","Type":"ContainerDied","Data":"e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a"}
Sep 18 20:52:10.223271 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:10.223236    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 18 20:52:15.223338 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:15.223292    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 18 20:52:20.223470 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:20.223425    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 18 20:52:20.223949 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:20.223541    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:52:25.222829 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:25.222790    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 18 20:52:30.223644 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:30.223596    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 18 20:52:35.223502 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:35.223454    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 18 20:52:36.196323 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.196297    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:52:36.304541 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.304509    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c923f5-2a76-4e89-a75e-7427dc033d16-proxy-tls\") pod \"35c923f5-2a76-4e89-a75e-7427dc033d16\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") "
Sep 18 20:52:36.305015 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.304563    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c923f5-2a76-4e89-a75e-7427dc033d16-kserve-provision-location\") pod \"35c923f5-2a76-4e89-a75e-7427dc033d16\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") "
Sep 18 20:52:36.305015 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.304592    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p82fm\" (UniqueName: \"kubernetes.io/projected/35c923f5-2a76-4e89-a75e-7427dc033d16-kube-api-access-p82fm\") pod \"35c923f5-2a76-4e89-a75e-7427dc033d16\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") "
Sep 18 20:52:36.305015 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.304625    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c923f5-2a76-4e89-a75e-7427dc033d16-kube-rbac-proxy-sar-config\") pod \"35c923f5-2a76-4e89-a75e-7427dc033d16\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") "
Sep 18 20:52:36.305015 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.304675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35c923f5-2a76-4e89-a75e-7427dc033d16-cabundle-cert\") pod \"35c923f5-2a76-4e89-a75e-7427dc033d16\" (UID: \"35c923f5-2a76-4e89-a75e-7427dc033d16\") "
Sep 18 20:52:36.305015 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.304963    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c923f5-2a76-4e89-a75e-7427dc033d16-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "35c923f5-2a76-4e89-a75e-7427dc033d16" (UID: "35c923f5-2a76-4e89-a75e-7427dc033d16"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:52:36.305257 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.305054    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c923f5-2a76-4e89-a75e-7427dc033d16-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35c923f5-2a76-4e89-a75e-7427dc033d16" (UID: "35c923f5-2a76-4e89-a75e-7427dc033d16"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:52:36.306927 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.306900    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c923f5-2a76-4e89-a75e-7427dc033d16-kube-api-access-p82fm" (OuterVolumeSpecName: "kube-api-access-p82fm") pod "35c923f5-2a76-4e89-a75e-7427dc033d16" (UID: "35c923f5-2a76-4e89-a75e-7427dc033d16"). InnerVolumeSpecName "kube-api-access-p82fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:52:36.307034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.306974    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c923f5-2a76-4e89-a75e-7427dc033d16-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35c923f5-2a76-4e89-a75e-7427dc033d16" (UID: "35c923f5-2a76-4e89-a75e-7427dc033d16"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:52:36.314230 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.314200    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c923f5-2a76-4e89-a75e-7427dc033d16-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35c923f5-2a76-4e89-a75e-7427dc033d16" (UID: "35c923f5-2a76-4e89-a75e-7427dc033d16"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:52:36.406212 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.406106    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35c923f5-2a76-4e89-a75e-7427dc033d16-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:52:36.406212 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.406170    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35c923f5-2a76-4e89-a75e-7427dc033d16-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:52:36.406212 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.406186    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p82fm\" (UniqueName: \"kubernetes.io/projected/35c923f5-2a76-4e89-a75e-7427dc033d16-kube-api-access-p82fm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:52:36.406212 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.406201    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35c923f5-2a76-4e89-a75e-7427dc033d16-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:52:36.406460 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.406217    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35c923f5-2a76-4e89-a75e-7427dc033d16-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:52:36.525687 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.525652    2583 generic.go:356] "Generic (PLEG): container finished" podID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerID="99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651" exitCode=137
Sep 18 20:52:36.525860 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.525711    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"
Sep 18 20:52:36.525860 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.525726    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" event={"ID":"35c923f5-2a76-4e89-a75e-7427dc033d16","Type":"ContainerDied","Data":"99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651"}
Sep 18 20:52:36.525860 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.525783    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx" event={"ID":"35c923f5-2a76-4e89-a75e-7427dc033d16","Type":"ContainerDied","Data":"6aee69bfd7879cafefa59e320ed4b8af957a5835b6d52565ec93a0de78aa79d7"}
Sep 18 20:52:36.525860 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.525808    2583 scope.go:117] "RemoveContainer" containerID="e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a"
Sep 18 20:52:36.534772 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.534753    2583 scope.go:117] "RemoveContainer" containerID="99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651"
Sep 18 20:52:36.542536 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.542519    2583 scope.go:117] "RemoveContainer" containerID="d96eff24c706a7205026b902d112ce40f208d6325073f8501c9f9907f064b441"
Sep 18 20:52:36.548710 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.548688    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"]
Sep 18 20:52:36.551525 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.551507    2583 scope.go:117] "RemoveContainer" containerID="e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a"
Sep 18 20:52:36.551802 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:52:36.551785    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a\": container with ID starting with e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a not found: ID does not exist" containerID="e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a"
Sep 18 20:52:36.551880 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.551814    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a"} err="failed to get container status \"e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a\": rpc error: code = NotFound desc = could not find container \"e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a\": container with ID starting with e90e597a80ade63a98ba5dc9d192baf1d8fe464c29f4cd99a801beb98a74195a not found: ID does not exist"
Sep 18 20:52:36.551880 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.551835    2583 scope.go:117] "RemoveContainer" containerID="99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651"
Sep 18 20:52:36.552137 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:52:36.552101    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651\": container with ID starting with 99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651 not found: ID does not exist" containerID="99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651"
Sep 18 20:52:36.552200 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.552143    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651"} err="failed to get container status \"99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651\": rpc error: code = NotFound desc = could not find container \"99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651\": container with ID starting with 99a7b05f2fe7277c03ccca363277f171f0cec4f4a24b17421c803f5084b23651 not found: ID does not exist"
Sep 18 20:52:36.552200 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.552162    2583 scope.go:117] "RemoveContainer" containerID="d96eff24c706a7205026b902d112ce40f208d6325073f8501c9f9907f064b441"
Sep 18 20:52:36.552444 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:52:36.552427    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96eff24c706a7205026b902d112ce40f208d6325073f8501c9f9907f064b441\": container with ID starting with d96eff24c706a7205026b902d112ce40f208d6325073f8501c9f9907f064b441 not found: ID does not exist" containerID="d96eff24c706a7205026b902d112ce40f208d6325073f8501c9f9907f064b441"
Sep 18 20:52:36.552493 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.552448    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96eff24c706a7205026b902d112ce40f208d6325073f8501c9f9907f064b441"} err="failed to get container status \"d96eff24c706a7205026b902d112ce40f208d6325073f8501c9f9907f064b441\": rpc error: code = NotFound desc = could not find container \"d96eff24c706a7205026b902d112ce40f208d6325073f8501c9f9907f064b441\": container with ID starting with d96eff24c706a7205026b902d112ce40f208d6325073f8501c9f9907f064b441 not found: ID does not exist"
Sep 18 20:52:36.552493 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:36.552465    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-mj6mx"]
Sep 18 20:52:37.315343 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:37.315312    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" path="/var/lib/kubelet/pods/35c923f5-2a76-4e89-a75e-7427dc033d16/volumes"
Sep 18 20:52:56.908780 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.908704    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"]
Sep 18 20:52:56.909174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909060    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kserve-container"
Sep 18 20:52:56.909174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909072    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kserve-container"
Sep 18 20:52:56.909174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909083    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container"
Sep 18 20:52:56.909174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909088    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container"
Sep 18 20:52:56.909174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909094    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kube-rbac-proxy"
Sep 18 20:52:56.909174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909099    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kube-rbac-proxy"
Sep 18 20:52:56.909174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909111    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kube-rbac-proxy"
Sep 18 20:52:56.909174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909116    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kube-rbac-proxy"
Sep 18 20:52:56.909174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909136    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="storage-initializer"
Sep 18 20:52:56.909174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909141    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="storage-initializer"
Sep 18 20:52:56.909174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909156    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="storage-initializer"
Sep 18 20:52:56.909174 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909161    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="storage-initializer"
Sep 18 20:52:56.909542 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909219    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kube-rbac-proxy"
Sep 18 20:52:56.909542 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909229    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kube-rbac-proxy"
Sep 18 20:52:56.909542 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909235    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6236e506-8c80-435d-8a0a-e0893d1ff712" containerName="kserve-container"
Sep 18 20:52:56.909542 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.909241    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="35c923f5-2a76-4e89-a75e-7427dc033d16" containerName="kserve-container"
Sep 18 20:52:56.913498 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.913477    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:56.916399 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.916375    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 18 20:52:56.916513 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.916456    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 18 20:52:56.925061 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:56.925040    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"]
Sep 18 20:52:57.085733 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.085699    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwb92\" (UniqueName: \"kubernetes.io/projected/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kube-api-access-rwb92\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.085890 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.085754    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.085890 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.085775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.085890 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.085808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.085890 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.085822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.186690 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.186658    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwb92\" (UniqueName: \"kubernetes.io/projected/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kube-api-access-rwb92\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.186867 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.186745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.186867 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.186782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.186867 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.186810    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.186867 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.186833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.187208 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.187187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.187550 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.187528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.187550 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.187545    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.189558 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.189540    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.195163 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.195141    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwb92\" (UniqueName: \"kubernetes.io/projected/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kube-api-access-rwb92\") pod \"isvc-triton-predictor-5d7db58676-8c4l7\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.224595 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.224571    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:52:57.500073 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.500043    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"]
Sep 18 20:52:57.502376 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:52:57.502348    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98ec1e9_74c0_43bc_a0e7_8a4d8626ba03.slice/crio-be541f90608e1e74434a89da9a91f46b1c3e35c82b084c4d46c57c1d316b555f WatchSource:0}: Error finding container be541f90608e1e74434a89da9a91f46b1c3e35c82b084c4d46c57c1d316b555f: Status 404 returned error can't find the container with id be541f90608e1e74434a89da9a91f46b1c3e35c82b084c4d46c57c1d316b555f
Sep 18 20:52:57.628071 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.628040    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" event={"ID":"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03","Type":"ContainerStarted","Data":"b8ffa3a14f103173c6240f9293d849e907b0c4a3a2598c31956ec3ffe9344a06"}
Sep 18 20:52:57.628284 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:52:57.628105    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" event={"ID":"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03","Type":"ContainerStarted","Data":"be541f90608e1e74434a89da9a91f46b1c3e35c82b084c4d46c57c1d316b555f"}
Sep 18 20:53:01.646627 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:01.646588    2583 generic.go:356] "Generic (PLEG): container finished" podID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerID="b8ffa3a14f103173c6240f9293d849e907b0c4a3a2598c31956ec3ffe9344a06" exitCode=0
Sep 18 20:53:01.647017 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:01.646664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" event={"ID":"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03","Type":"ContainerDied","Data":"b8ffa3a14f103173c6240f9293d849e907b0c4a3a2598c31956ec3ffe9344a06"}
Sep 18 20:53:23.282532 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:23.281720    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"]
Sep 18 20:53:23.282532 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:23.282058    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container" containerID="cri-o://c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c" gracePeriod=30
Sep 18 20:53:23.282532 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:23.282287    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kube-rbac-proxy" containerID="cri-o://c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573" gracePeriod=30
Sep 18 20:53:23.817552 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:23.817505    2583 generic.go:356] "Generic (PLEG): container finished" podID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerID="c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573" exitCode=2
Sep 18 20:53:23.817742 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:23.817586    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" event={"ID":"a8302771-4bc6-4e38-b6a0-eff6e523f3c4","Type":"ContainerDied","Data":"c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573"}
Sep 18 20:53:25.888178 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:25.888102    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 18 20:53:25.893606 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:25.893566    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:53:28.573806 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.573765    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"]
Sep 18 20:53:28.586646 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.586614    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.587606 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.587583    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"]
Sep 18 20:53:28.590419 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.589542    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 20:53:28.590419 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.589789    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 20:53:28.713544 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.713510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a692b8b8-ae99-4167-a770-8f298753ab15-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.713544 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.713548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a692b8b8-ae99-4167-a770-8f298753ab15-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.713977 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.713579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2s2w\" (UniqueName: \"kubernetes.io/projected/a692b8b8-ae99-4167-a770-8f298753ab15-kube-api-access-z2s2w\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.713977 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.713653    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a692b8b8-ae99-4167-a770-8f298753ab15-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.713977 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.713697    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a692b8b8-ae99-4167-a770-8f298753ab15-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.815098 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.815012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a692b8b8-ae99-4167-a770-8f298753ab15-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.815098 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.815066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2s2w\" (UniqueName: \"kubernetes.io/projected/a692b8b8-ae99-4167-a770-8f298753ab15-kube-api-access-z2s2w\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.815098 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.815094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a692b8b8-ae99-4167-a770-8f298753ab15-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.815603 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.815143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a692b8b8-ae99-4167-a770-8f298753ab15-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.815603 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.815283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a692b8b8-ae99-4167-a770-8f298753ab15-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.815603 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:53:28.815408    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 18 20:53:28.815603 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:53:28.815472    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a692b8b8-ae99-4167-a770-8f298753ab15-proxy-tls podName:a692b8b8-ae99-4167-a770-8f298753ab15 nodeName:}" failed. No retries permitted until 2026-09-18 20:53:29.31545475 +0000 UTC m=+1960.223357007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a692b8b8-ae99-4167-a770-8f298753ab15-proxy-tls") pod "isvc-canary-rollback-predictor-7f7b644c9-d2cnk" (UID: "a692b8b8-ae99-4167-a770-8f298753ab15") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 18 20:53:28.815893 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.815827    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a692b8b8-ae99-4167-a770-8f298753ab15-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.816111 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.816091    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a692b8b8-ae99-4167-a770-8f298753ab15-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.816589 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.816562    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a692b8b8-ae99-4167-a770-8f298753ab15-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:28.825636 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:28.825607    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2s2w\" (UniqueName: \"kubernetes.io/projected/a692b8b8-ae99-4167-a770-8f298753ab15-kube-api-access-z2s2w\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:29.309547 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.309520    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:53:29.319726 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.319700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a692b8b8-ae99-4167-a770-8f298753ab15-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:29.323037 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.322992    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a692b8b8-ae99-4167-a770-8f298753ab15-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-d2cnk\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:29.420925 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.420895    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-cabundle-cert\") pod \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") "
Sep 18 20:53:29.421100 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.420940    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-proxy-tls\") pod \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") "
Sep 18 20:53:29.421100 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.421000    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5gks\" (UniqueName: \"kubernetes.io/projected/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kube-api-access-g5gks\") pod \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") "
Sep 18 20:53:29.421100 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.421068    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kube-rbac-proxy-sar-config\") pod \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") "
Sep 18 20:53:29.421310 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.421109    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kserve-provision-location\") pod \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\" (UID: \"a8302771-4bc6-4e38-b6a0-eff6e523f3c4\") "
Sep 18 20:53:29.421367 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.421345    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8302771-4bc6-4e38-b6a0-eff6e523f3c4" (UID: "a8302771-4bc6-4e38-b6a0-eff6e523f3c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:53:29.421733 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.421457    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:53:29.421733 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.421697    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8302771-4bc6-4e38-b6a0-eff6e523f3c4" (UID: "a8302771-4bc6-4e38-b6a0-eff6e523f3c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:53:29.421905 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.421735    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8302771-4bc6-4e38-b6a0-eff6e523f3c4" (UID: "a8302771-4bc6-4e38-b6a0-eff6e523f3c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:53:29.424243 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.424209    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kube-api-access-g5gks" (OuterVolumeSpecName: "kube-api-access-g5gks") pod "a8302771-4bc6-4e38-b6a0-eff6e523f3c4" (UID: "a8302771-4bc6-4e38-b6a0-eff6e523f3c4"). InnerVolumeSpecName "kube-api-access-g5gks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:53:29.424342 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.424266    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8302771-4bc6-4e38-b6a0-eff6e523f3c4" (UID: "a8302771-4bc6-4e38-b6a0-eff6e523f3c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:53:29.502494 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.502418    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:29.522323 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.522293    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:53:29.522323 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.522326    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5gks\" (UniqueName: \"kubernetes.io/projected/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kube-api-access-g5gks\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:53:29.522590 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.522340    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:53:29.522590 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.522354    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8302771-4bc6-4e38-b6a0-eff6e523f3c4-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:53:29.814050 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.814023    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"]
Sep 18 20:53:29.816171 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:53:29.816112    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda692b8b8_ae99_4167_a770_8f298753ab15.slice/crio-ee2a4371a1537cd4735b634519a3d187ff9f786a432e588945ebacfba7838eb6 WatchSource:0}: Error finding container ee2a4371a1537cd4735b634519a3d187ff9f786a432e588945ebacfba7838eb6: Status 404 returned error can't find the container with id ee2a4371a1537cd4735b634519a3d187ff9f786a432e588945ebacfba7838eb6
Sep 18 20:53:29.858698 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.858662    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" event={"ID":"a692b8b8-ae99-4167-a770-8f298753ab15","Type":"ContainerStarted","Data":"ee2a4371a1537cd4735b634519a3d187ff9f786a432e588945ebacfba7838eb6"}
Sep 18 20:53:29.862012 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.861985    2583 generic.go:356] "Generic (PLEG): container finished" podID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerID="c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c" exitCode=0
Sep 18 20:53:29.862146 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.862054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" event={"ID":"a8302771-4bc6-4e38-b6a0-eff6e523f3c4","Type":"ContainerDied","Data":"c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c"}
Sep 18 20:53:29.862146 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.862101    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"
Sep 18 20:53:29.862146 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.862112    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg" event={"ID":"a8302771-4bc6-4e38-b6a0-eff6e523f3c4","Type":"ContainerDied","Data":"a95bc1eb536844871b9661e765019dc7bc66563e812c29dec44f8c337a640345"}
Sep 18 20:53:29.862326 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.862171    2583 scope.go:117] "RemoveContainer" containerID="c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573"
Sep 18 20:53:29.881026 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.881001    2583 scope.go:117] "RemoveContainer" containerID="c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c"
Sep 18 20:53:29.896030 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.896009    2583 scope.go:117] "RemoveContainer" containerID="826c53c32e50e32064badbc4a6dbb9a3a499713a034b4d60aca9e54e30f1dc51"
Sep 18 20:53:29.902788 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.902765    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"]
Sep 18 20:53:29.907359 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.907338    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-xrddg"]
Sep 18 20:53:29.912425 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.912404    2583 scope.go:117] "RemoveContainer" containerID="c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573"
Sep 18 20:53:29.912887 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:53:29.912812    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573\": container with ID starting with c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573 not found: ID does not exist" containerID="c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573"
Sep 18 20:53:29.912990 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.912887    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573"} err="failed to get container status \"c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573\": rpc error: code = NotFound desc = could not find container \"c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573\": container with ID starting with c785c9649bf37f5e77b98204074cc09522b33092d11401f795e030e2434a4573 not found: ID does not exist"
Sep 18 20:53:29.912990 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.912915    2583 scope.go:117] "RemoveContainer" containerID="c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c"
Sep 18 20:53:29.913300 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:53:29.913268    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c\": container with ID starting with c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c not found: ID does not exist" containerID="c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c"
Sep 18 20:53:29.913376 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.913308    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c"} err="failed to get container status \"c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c\": rpc error: code = NotFound desc = could not find container \"c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c\": container with ID starting with c1b7eb46cabb412b2fcc17d517963399fb8b4e433d12d9fdf3092dfcabba420c not found: ID does not exist"
Sep 18 20:53:29.913376 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.913328    2583 scope.go:117] "RemoveContainer" containerID="826c53c32e50e32064badbc4a6dbb9a3a499713a034b4d60aca9e54e30f1dc51"
Sep 18 20:53:29.913740 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:53:29.913679    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826c53c32e50e32064badbc4a6dbb9a3a499713a034b4d60aca9e54e30f1dc51\": container with ID starting with 826c53c32e50e32064badbc4a6dbb9a3a499713a034b4d60aca9e54e30f1dc51 not found: ID does not exist" containerID="826c53c32e50e32064badbc4a6dbb9a3a499713a034b4d60aca9e54e30f1dc51"
Sep 18 20:53:29.913740 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:29.913712    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826c53c32e50e32064badbc4a6dbb9a3a499713a034b4d60aca9e54e30f1dc51"} err="failed to get container status \"826c53c32e50e32064badbc4a6dbb9a3a499713a034b4d60aca9e54e30f1dc51\": rpc error: code = NotFound desc = could not find container \"826c53c32e50e32064badbc4a6dbb9a3a499713a034b4d60aca9e54e30f1dc51\": container with ID starting with 826c53c32e50e32064badbc4a6dbb9a3a499713a034b4d60aca9e54e30f1dc51 not found: ID does not exist"
Sep 18 20:53:30.874098 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:30.874056    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" event={"ID":"a692b8b8-ae99-4167-a770-8f298753ab15","Type":"ContainerStarted","Data":"9cc5180837f491a400521d4d6965a0a000a6d61faeaeb9dbdb52cbb89042229b"}
Sep 18 20:53:31.324163 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:31.324111    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" path="/var/lib/kubelet/pods/a8302771-4bc6-4e38-b6a0-eff6e523f3c4/volumes"
Sep 18 20:53:34.905067 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:34.905028    2583 generic.go:356] "Generic (PLEG): container finished" podID="a692b8b8-ae99-4167-a770-8f298753ab15" containerID="9cc5180837f491a400521d4d6965a0a000a6d61faeaeb9dbdb52cbb89042229b" exitCode=0
Sep 18 20:53:34.905680 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:34.905233    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" event={"ID":"a692b8b8-ae99-4167-a770-8f298753ab15","Type":"ContainerDied","Data":"9cc5180837f491a400521d4d6965a0a000a6d61faeaeb9dbdb52cbb89042229b"}
Sep 18 20:53:35.914625 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:35.914590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" event={"ID":"a692b8b8-ae99-4167-a770-8f298753ab15","Type":"ContainerStarted","Data":"728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297"}
Sep 18 20:53:35.915102 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:35.914650    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" event={"ID":"a692b8b8-ae99-4167-a770-8f298753ab15","Type":"ContainerStarted","Data":"89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7"}
Sep 18 20:53:35.915102 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:35.914925    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:35.915258 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:35.915189    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:35.916530 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:35.916500    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:53:35.934945 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:35.934898    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podStartSLOduration=3.557957524 podStartE2EDuration="7.934884294s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.37692677s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:53:35.93343288 +0000 UTC m=+1966.841335159" watchObservedRunningTime="2026-09-18 20:53:35.934884294 +0000 UTC m=+1966.842786574"
Sep 18 20:53:36.921233 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:36.921186    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:53:41.926829 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:41.926725    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:53:41.927481 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:41.927444    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:53:51.928191 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:53:51.928142    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:54:01.927927 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:01.927872    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:54:11.927879 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:11.927829    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:54:21.927994 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:21.927948    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:54:31.928211 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:31.928162    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:54:41.928014 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:41.927969    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:54:43.310985 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:43.310946    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:54:51.822458 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:51.822427    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"]
Sep 18 20:54:51.823043 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:51.822775    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" containerID="cri-o://89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7" gracePeriod=30
Sep 18 20:54:51.823043 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:51.822830    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kube-rbac-proxy" containerID="cri-o://728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297" gracePeriod=30
Sep 18 20:54:51.823206 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:51.823155    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:54:51.921848 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:51.921806    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 18 20:54:55.352555 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:55.352522    2583 generic.go:356] "Generic (PLEG): container finished" podID="a692b8b8-ae99-4167-a770-8f298753ab15" containerID="728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297" exitCode=2
Sep 18 20:54:55.352882 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:55.352589    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" event={"ID":"a692b8b8-ae99-4167-a770-8f298753ab15","Type":"ContainerDied","Data":"728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297"}
Sep 18 20:54:56.922146 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:56.922084    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 18 20:54:57.370354 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:57.370281    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" event={"ID":"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03","Type":"ContainerStarted","Data":"95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7"}
Sep 18 20:54:57.370354 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:57.370335    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" event={"ID":"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03","Type":"ContainerStarted","Data":"a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b"}
Sep 18 20:54:57.370602 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:57.370471    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:54:57.389719 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:57.389679    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" podStartSLOduration=2.658436798 podStartE2EDuration="2m1.389666388s" totalImagesPullingTime="1m54.824662359s" totalInitContainerRuntime="3.906567231s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:54:57.388112825 +0000 UTC m=+2048.296015106" watchObservedRunningTime="2026-09-18 20:54:57.389666388 +0000 UTC m=+2048.297568667"
Sep 18 20:54:58.376175 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.376143    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:54:58.377264 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.377238    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:54:58.601826 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.601795    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"]
Sep 18 20:54:58.602256 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.602241    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container"
Sep 18 20:54:58.602309 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.602258    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container"
Sep 18 20:54:58.602309 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.602270    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="storage-initializer"
Sep 18 20:54:58.602309 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.602276    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="storage-initializer"
Sep 18 20:54:58.602309 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.602288    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kube-rbac-proxy"
Sep 18 20:54:58.602309 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.602297    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kube-rbac-proxy"
Sep 18 20:54:58.602466 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.602373    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kserve-container"
Sep 18 20:54:58.602466 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.602388    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8302771-4bc6-4e38-b6a0-eff6e523f3c4" containerName="kube-rbac-proxy"
Sep 18 20:54:58.634998 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.634929    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"]
Sep 18 20:54:58.635152 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.635030    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.650325 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.650302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ce61be2-5593-41b3-8197-b45e37b11a02-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.650452 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.650357    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce61be2-5593-41b3-8197-b45e37b11a02-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.650452 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.650381    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce61be2-5593-41b3-8197-b45e37b11a02-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.650452 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.650409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce61be2-5593-41b3-8197-b45e37b11a02-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.650645 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.650477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlgwz\" (UniqueName: \"kubernetes.io/projected/8ce61be2-5593-41b3-8197-b45e37b11a02-kube-api-access-rlgwz\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.751462 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.751426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce61be2-5593-41b3-8197-b45e37b11a02-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.751635 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.751480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlgwz\" (UniqueName: \"kubernetes.io/projected/8ce61be2-5593-41b3-8197-b45e37b11a02-kube-api-access-rlgwz\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.751635 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.751539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ce61be2-5593-41b3-8197-b45e37b11a02-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.751635 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.751600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce61be2-5593-41b3-8197-b45e37b11a02-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.751635 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.751631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce61be2-5593-41b3-8197-b45e37b11a02-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.752017 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.751991    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce61be2-5593-41b3-8197-b45e37b11a02-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.752306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.752287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ce61be2-5593-41b3-8197-b45e37b11a02-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.752374 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.752289    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce61be2-5593-41b3-8197-b45e37b11a02-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.754266 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.754244    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce61be2-5593-41b3-8197-b45e37b11a02-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.759642 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.759618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlgwz\" (UniqueName: \"kubernetes.io/projected/8ce61be2-5593-41b3-8197-b45e37b11a02-kube-api-access-rlgwz\") pod \"isvc-canary-rollback-predictor-7f7b644c9-88bbv\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:58.945267 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:58.945232    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:54:59.232217 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:59.232184    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"]
Sep 18 20:54:59.234181 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:54:59.234150    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce61be2_5593_41b3_8197_b45e37b11a02.slice/crio-196148a05097101cf82e14246c60c785bef4d7b5be5c534ee87600ad6bc18681 WatchSource:0}: Error finding container 196148a05097101cf82e14246c60c785bef4d7b5be5c534ee87600ad6bc18681: Status 404 returned error can't find the container with id 196148a05097101cf82e14246c60c785bef4d7b5be5c534ee87600ad6bc18681
Sep 18 20:54:59.384274 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:59.384232    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" event={"ID":"8ce61be2-5593-41b3-8197-b45e37b11a02","Type":"ContainerStarted","Data":"7586e41a8eb68a56aa91d7aa5523e681430dfa72648f265efbff9c037f242b8a"}
Sep 18 20:54:59.384672 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:59.384306    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" event={"ID":"8ce61be2-5593-41b3-8197-b45e37b11a02","Type":"ContainerStarted","Data":"196148a05097101cf82e14246c60c785bef4d7b5be5c534ee87600ad6bc18681"}
Sep 18 20:54:59.384672 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:54:59.384621    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:55:01.823739 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:01.823645    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:55:01.922206 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:01.922168    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 18 20:55:01.922366 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:01.922279    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:55:03.406890 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:03.406859    2583 generic.go:356] "Generic (PLEG): container finished" podID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerID="7586e41a8eb68a56aa91d7aa5523e681430dfa72648f265efbff9c037f242b8a" exitCode=0
Sep 18 20:55:03.407263 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:03.406911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" event={"ID":"8ce61be2-5593-41b3-8197-b45e37b11a02","Type":"ContainerDied","Data":"7586e41a8eb68a56aa91d7aa5523e681430dfa72648f265efbff9c037f242b8a"}
Sep 18 20:55:04.390237 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:04.390210    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:55:04.390698 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:04.390670    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:55:04.416747 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:04.416715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" event={"ID":"8ce61be2-5593-41b3-8197-b45e37b11a02","Type":"ContainerStarted","Data":"b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15"}
Sep 18 20:55:04.417168 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:04.416804    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" event={"ID":"8ce61be2-5593-41b3-8197-b45e37b11a02","Type":"ContainerStarted","Data":"1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651"}
Sep 18 20:55:04.417496 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:04.417477    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:55:04.417589 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:04.417528    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:55:04.418921 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:04.418893    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:55:04.435917 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:04.435862    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podStartSLOduration=2.485899089 podStartE2EDuration="6.435847357s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.949948268s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:55:04.434276874 +0000 UTC m=+2055.342179149" watchObservedRunningTime="2026-09-18 20:55:04.435847357 +0000 UTC m=+2055.343749679"
Sep 18 20:55:05.421903 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:05.421868    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:55:06.922074 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:06.922035    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 18 20:55:10.426478 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:10.426447    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:55:10.426971 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:10.426942    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:55:11.824347 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:11.824314    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:55:11.921969 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:11.921923    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 18 20:55:14.391873 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:14.391848    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:55:16.921265 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:16.921225    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 18 20:55:20.427704 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:20.427656    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:55:21.352034 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:21.351995    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"]
Sep 18 20:55:21.352335 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:21.352305    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="kserve-container" containerID="cri-o://a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b" gracePeriod=30
Sep 18 20:55:21.352466 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:21.352342    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="kube-rbac-proxy" containerID="cri-o://95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7" gracePeriod=30
Sep 18 20:55:21.499897 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:21.499866    2583 generic.go:356] "Generic (PLEG): container finished" podID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerID="95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7" exitCode=2
Sep 18 20:55:21.500277 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:21.499917    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" event={"ID":"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03","Type":"ContainerDied","Data":"95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7"}
Sep 18 20:55:21.921526 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:21.921488    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 18 20:55:22.499005 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.498971    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:55:22.506950 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.506924    2583 generic.go:356] "Generic (PLEG): container finished" podID="a692b8b8-ae99-4167-a770-8f298753ab15" containerID="89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7" exitCode=137
Sep 18 20:55:22.507301 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.506982    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"
Sep 18 20:55:22.507301 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.507017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" event={"ID":"a692b8b8-ae99-4167-a770-8f298753ab15","Type":"ContainerDied","Data":"89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7"}
Sep 18 20:55:22.507301 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.507082    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk" event={"ID":"a692b8b8-ae99-4167-a770-8f298753ab15","Type":"ContainerDied","Data":"ee2a4371a1537cd4735b634519a3d187ff9f786a432e588945ebacfba7838eb6"}
Sep 18 20:55:22.507301 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.507108    2583 scope.go:117] "RemoveContainer" containerID="728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297"
Sep 18 20:55:22.515547 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.515529    2583 scope.go:117] "RemoveContainer" containerID="89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7"
Sep 18 20:55:22.528846 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.528727    2583 scope.go:117] "RemoveContainer" containerID="9cc5180837f491a400521d4d6965a0a000a6d61faeaeb9dbdb52cbb89042229b"
Sep 18 20:55:22.538690 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.538672    2583 scope.go:117] "RemoveContainer" containerID="728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297"
Sep 18 20:55:22.539004 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:55:22.538986    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297\": container with ID starting with 728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297 not found: ID does not exist" containerID="728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297"
Sep 18 20:55:22.539057 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.539015    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297"} err="failed to get container status \"728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297\": rpc error: code = NotFound desc = could not find container \"728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297\": container with ID starting with 728104e806d22f0bf4aeb9f69dd0d0cf69592cc5fb41d9775238fc8ae21fd297 not found: ID does not exist"
Sep 18 20:55:22.539057 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.539032    2583 scope.go:117] "RemoveContainer" containerID="89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7"
Sep 18 20:55:22.539358 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:55:22.539331    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7\": container with ID starting with 89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7 not found: ID does not exist" containerID="89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7"
Sep 18 20:55:22.539418 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.539368    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7"} err="failed to get container status \"89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7\": rpc error: code = NotFound desc = could not find container \"89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7\": container with ID starting with 89bc8ae1a13171521e34a3cda77d6c83ed8513e88faa76b8665c1289a55d55e7 not found: ID does not exist"
Sep 18 20:55:22.539418 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.539406    2583 scope.go:117] "RemoveContainer" containerID="9cc5180837f491a400521d4d6965a0a000a6d61faeaeb9dbdb52cbb89042229b"
Sep 18 20:55:22.539697 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:55:22.539677    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc5180837f491a400521d4d6965a0a000a6d61faeaeb9dbdb52cbb89042229b\": container with ID starting with 9cc5180837f491a400521d4d6965a0a000a6d61faeaeb9dbdb52cbb89042229b not found: ID does not exist" containerID="9cc5180837f491a400521d4d6965a0a000a6d61faeaeb9dbdb52cbb89042229b"
Sep 18 20:55:22.539739 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.539706    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc5180837f491a400521d4d6965a0a000a6d61faeaeb9dbdb52cbb89042229b"} err="failed to get container status \"9cc5180837f491a400521d4d6965a0a000a6d61faeaeb9dbdb52cbb89042229b\": rpc error: code = NotFound desc = could not find container \"9cc5180837f491a400521d4d6965a0a000a6d61faeaeb9dbdb52cbb89042229b\": container with ID starting with 9cc5180837f491a400521d4d6965a0a000a6d61faeaeb9dbdb52cbb89042229b not found: ID does not exist"
Sep 18 20:55:22.555033 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.555013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a692b8b8-ae99-4167-a770-8f298753ab15-proxy-tls\") pod \"a692b8b8-ae99-4167-a770-8f298753ab15\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") "
Sep 18 20:55:22.555143 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.555055    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a692b8b8-ae99-4167-a770-8f298753ab15-kube-rbac-proxy-sar-config\") pod \"a692b8b8-ae99-4167-a770-8f298753ab15\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") "
Sep 18 20:55:22.555143 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.555080    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a692b8b8-ae99-4167-a770-8f298753ab15-kserve-provision-location\") pod \"a692b8b8-ae99-4167-a770-8f298753ab15\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") "
Sep 18 20:55:22.555143 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.555099    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a692b8b8-ae99-4167-a770-8f298753ab15-cabundle-cert\") pod \"a692b8b8-ae99-4167-a770-8f298753ab15\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") "
Sep 18 20:55:22.555312 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.555163    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2s2w\" (UniqueName: \"kubernetes.io/projected/a692b8b8-ae99-4167-a770-8f298753ab15-kube-api-access-z2s2w\") pod \"a692b8b8-ae99-4167-a770-8f298753ab15\" (UID: \"a692b8b8-ae99-4167-a770-8f298753ab15\") "
Sep 18 20:55:22.555438 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.555412    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a692b8b8-ae99-4167-a770-8f298753ab15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a692b8b8-ae99-4167-a770-8f298753ab15" (UID: "a692b8b8-ae99-4167-a770-8f298753ab15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:55:22.555523 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.555470    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a692b8b8-ae99-4167-a770-8f298753ab15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a692b8b8-ae99-4167-a770-8f298753ab15" (UID: "a692b8b8-ae99-4167-a770-8f298753ab15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:55:22.555523 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.555474    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a692b8b8-ae99-4167-a770-8f298753ab15-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a692b8b8-ae99-4167-a770-8f298753ab15" (UID: "a692b8b8-ae99-4167-a770-8f298753ab15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:55:22.557330 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.557309    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a692b8b8-ae99-4167-a770-8f298753ab15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a692b8b8-ae99-4167-a770-8f298753ab15" (UID: "a692b8b8-ae99-4167-a770-8f298753ab15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:55:22.557449 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.557430    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a692b8b8-ae99-4167-a770-8f298753ab15-kube-api-access-z2s2w" (OuterVolumeSpecName: "kube-api-access-z2s2w") pod "a692b8b8-ae99-4167-a770-8f298753ab15" (UID: "a692b8b8-ae99-4167-a770-8f298753ab15"). InnerVolumeSpecName "kube-api-access-z2s2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:55:22.656192 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.656097    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2s2w\" (UniqueName: \"kubernetes.io/projected/a692b8b8-ae99-4167-a770-8f298753ab15-kube-api-access-z2s2w\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:55:22.656192 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.656158    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a692b8b8-ae99-4167-a770-8f298753ab15-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:55:22.656192 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.656173    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a692b8b8-ae99-4167-a770-8f298753ab15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:55:22.656192 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.656181    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a692b8b8-ae99-4167-a770-8f298753ab15-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:55:22.656192 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.656190    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a692b8b8-ae99-4167-a770-8f298753ab15-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:55:22.833338 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.833298    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"]
Sep 18 20:55:22.836054 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:22.836027    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-d2cnk"]
Sep 18 20:55:23.316188 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:23.316157    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" path="/var/lib/kubelet/pods/a692b8b8-ae99-4167-a770-8f298753ab15/volumes"
Sep 18 20:55:23.989313 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:23.989255    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:55:24.069967 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.069933    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-proxy-tls\") pod \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") "
Sep 18 20:55:24.070175 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.069980    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kube-rbac-proxy-sar-config\") pod \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") "
Sep 18 20:55:24.070175 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.070011    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwb92\" (UniqueName: \"kubernetes.io/projected/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kube-api-access-rwb92\") pod \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") "
Sep 18 20:55:24.070175 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.070151    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kserve-provision-location\") pod \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") "
Sep 18 20:55:24.070357 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.070182    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-cabundle-cert\") pod \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\" (UID: \"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03\") "
Sep 18 20:55:24.070517 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.070407    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" (UID: "a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:55:24.070572 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.070521    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" (UID: "a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:55:24.070572 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.070540    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" (UID: "a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:55:24.072378 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.072353    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kube-api-access-rwb92" (OuterVolumeSpecName: "kube-api-access-rwb92") pod "a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" (UID: "a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03"). InnerVolumeSpecName "kube-api-access-rwb92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:55:24.072378 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.072369    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" (UID: "a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:55:24.171213 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.171175    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwb92\" (UniqueName: \"kubernetes.io/projected/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kube-api-access-rwb92\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:55:24.171213 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.171206    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:55:24.171213 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.171216    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:55:24.171437 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.171226    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:55:24.171437 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.171234    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:55:24.520066 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.520032    2583 generic.go:356] "Generic (PLEG): container finished" podID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerID="a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b" exitCode=0
Sep 18 20:55:24.520349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.520112    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"
Sep 18 20:55:24.520349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.520110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" event={"ID":"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03","Type":"ContainerDied","Data":"a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b"}
Sep 18 20:55:24.520349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.520266    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7" event={"ID":"a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03","Type":"ContainerDied","Data":"be541f90608e1e74434a89da9a91f46b1c3e35c82b084c4d46c57c1d316b555f"}
Sep 18 20:55:24.520349 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.520293    2583 scope.go:117] "RemoveContainer" containerID="95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7"
Sep 18 20:55:24.529871 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.529854    2583 scope.go:117] "RemoveContainer" containerID="a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b"
Sep 18 20:55:24.538028 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.538013    2583 scope.go:117] "RemoveContainer" containerID="b8ffa3a14f103173c6240f9293d849e907b0c4a3a2598c31956ec3ffe9344a06"
Sep 18 20:55:24.543196 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.543176    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"]
Sep 18 20:55:24.546311 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.546290    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8c4l7"]
Sep 18 20:55:24.547275 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.547261    2583 scope.go:117] "RemoveContainer" containerID="95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7"
Sep 18 20:55:24.547550 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:55:24.547533    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7\": container with ID starting with 95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7 not found: ID does not exist" containerID="95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7"
Sep 18 20:55:24.547595 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.547566    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7"} err="failed to get container status \"95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7\": rpc error: code = NotFound desc = could not find container \"95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7\": container with ID starting with 95c9c55c69d024e0767cfae4f5ca0af97049338f643bec47e76dda97398038f7 not found: ID does not exist"
Sep 18 20:55:24.547595 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.547583    2583 scope.go:117] "RemoveContainer" containerID="a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b"
Sep 18 20:55:24.547847 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:55:24.547828    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b\": container with ID starting with a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b not found: ID does not exist" containerID="a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b"
Sep 18 20:55:24.547903 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.547853    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b"} err="failed to get container status \"a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b\": rpc error: code = NotFound desc = could not find container \"a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b\": container with ID starting with a8b10cdcb28b5d008e79c4aa72d6a2752f977a1250d5cdca42efa031d011317b not found: ID does not exist"
Sep 18 20:55:24.547903 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.547868    2583 scope.go:117] "RemoveContainer" containerID="b8ffa3a14f103173c6240f9293d849e907b0c4a3a2598c31956ec3ffe9344a06"
Sep 18 20:55:24.548148 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:55:24.548106    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ffa3a14f103173c6240f9293d849e907b0c4a3a2598c31956ec3ffe9344a06\": container with ID starting with b8ffa3a14f103173c6240f9293d849e907b0c4a3a2598c31956ec3ffe9344a06 not found: ID does not exist" containerID="b8ffa3a14f103173c6240f9293d849e907b0c4a3a2598c31956ec3ffe9344a06"
Sep 18 20:55:24.548243 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:24.548225    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ffa3a14f103173c6240f9293d849e907b0c4a3a2598c31956ec3ffe9344a06"} err="failed to get container status \"b8ffa3a14f103173c6240f9293d849e907b0c4a3a2598c31956ec3ffe9344a06\": rpc error: code = NotFound desc = could not find container \"b8ffa3a14f103173c6240f9293d849e907b0c4a3a2598c31956ec3ffe9344a06\": container with ID starting with b8ffa3a14f103173c6240f9293d849e907b0c4a3a2598c31956ec3ffe9344a06 not found: ID does not exist"
Sep 18 20:55:25.316763 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:25.316728    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" path="/var/lib/kubelet/pods/a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03/volumes"
Sep 18 20:55:26.557245 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557209    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"]
Sep 18 20:55:26.557612 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557554    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="storage-initializer"
Sep 18 20:55:26.557612 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557565    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="storage-initializer"
Sep 18 20:55:26.557612 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557584    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="kube-rbac-proxy"
Sep 18 20:55:26.557612 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557590    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="kube-rbac-proxy"
Sep 18 20:55:26.557612 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557600    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="storage-initializer"
Sep 18 20:55:26.557612 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557605    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="storage-initializer"
Sep 18 20:55:26.557612 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557612    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kube-rbac-proxy"
Sep 18 20:55:26.557612 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557617    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kube-rbac-proxy"
Sep 18 20:55:26.557918 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557626    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="kserve-container"
Sep 18 20:55:26.557918 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557631    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="kserve-container"
Sep 18 20:55:26.557918 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557638    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container"
Sep 18 20:55:26.557918 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557643    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container"
Sep 18 20:55:26.557918 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557696    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kserve-container"
Sep 18 20:55:26.557918 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557705    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a692b8b8-ae99-4167-a770-8f298753ab15" containerName="kube-rbac-proxy"
Sep 18 20:55:26.557918 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557711    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="kserve-container"
Sep 18 20:55:26.557918 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.557717    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98ec1e9-74c0-43bc-a0e7-8a4d8626ba03" containerName="kube-rbac-proxy"
Sep 18 20:55:26.588339 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.588309    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"]
Sep 18 20:55:26.588505 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.588359    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.591290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.591268    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 18 20:55:26.591415 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.591312    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 18 20:55:26.691371 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.691338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c262d8ce-d59b-4222-a950-101197b5a722-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.691371 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.691370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c262d8ce-d59b-4222-a950-101197b5a722-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.691592 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.691415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c262d8ce-d59b-4222-a950-101197b5a722-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.691592 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.691438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c262d8ce-d59b-4222-a950-101197b5a722-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.691592 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.691469    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbc44\" (UniqueName: \"kubernetes.io/projected/c262d8ce-d59b-4222-a950-101197b5a722-kube-api-access-dbc44\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.792008 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.791967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c262d8ce-d59b-4222-a950-101197b5a722-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.792008 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.792010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c262d8ce-d59b-4222-a950-101197b5a722-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.792244 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.792046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c262d8ce-d59b-4222-a950-101197b5a722-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.792244 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.792062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c262d8ce-d59b-4222-a950-101197b5a722-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.792244 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.792099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbc44\" (UniqueName: \"kubernetes.io/projected/c262d8ce-d59b-4222-a950-101197b5a722-kube-api-access-dbc44\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.792456 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.792435    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c262d8ce-d59b-4222-a950-101197b5a722-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.792793 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.792770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c262d8ce-d59b-4222-a950-101197b5a722-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.792896 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.792769    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c262d8ce-d59b-4222-a950-101197b5a722-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.794630 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.794609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c262d8ce-d59b-4222-a950-101197b5a722-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.800592 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.800568    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbc44\" (UniqueName: \"kubernetes.io/projected/c262d8ce-d59b-4222-a950-101197b5a722-kube-api-access-dbc44\") pod \"isvc-xgboost-predictor-7566d65bf7-wvnhw\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:26.898413 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:26.898350    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:27.206019 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:27.205993    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"]
Sep 18 20:55:27.208082 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:55:27.208050    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc262d8ce_d59b_4222_a950_101197b5a722.slice/crio-ab776f6c7ac88284f9e747ff9742a0ae5bf7fc53724357ffe949a71e41d9a25f WatchSource:0}: Error finding container ab776f6c7ac88284f9e747ff9742a0ae5bf7fc53724357ffe949a71e41d9a25f: Status 404 returned error can't find the container with id ab776f6c7ac88284f9e747ff9742a0ae5bf7fc53724357ffe949a71e41d9a25f
Sep 18 20:55:27.537680 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:27.537597    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" event={"ID":"c262d8ce-d59b-4222-a950-101197b5a722","Type":"ContainerStarted","Data":"d8517e104268ae1b9fc443da9d943e0fa7f1b675484068d449814116c475918c"}
Sep 18 20:55:27.537680 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:27.537669    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" event={"ID":"c262d8ce-d59b-4222-a950-101197b5a722","Type":"ContainerStarted","Data":"ab776f6c7ac88284f9e747ff9742a0ae5bf7fc53724357ffe949a71e41d9a25f"}
Sep 18 20:55:30.427010 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:30.426975    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:55:31.557763 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:31.557687    2583 generic.go:356] "Generic (PLEG): container finished" podID="c262d8ce-d59b-4222-a950-101197b5a722" containerID="d8517e104268ae1b9fc443da9d943e0fa7f1b675484068d449814116c475918c" exitCode=0
Sep 18 20:55:31.557763 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:31.557752    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" event={"ID":"c262d8ce-d59b-4222-a950-101197b5a722","Type":"ContainerDied","Data":"d8517e104268ae1b9fc443da9d943e0fa7f1b675484068d449814116c475918c"}
Sep 18 20:55:31.559150 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:31.559114    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:55:40.427470 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:40.427410    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:55:50.427545 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:50.427493    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:55:51.405632 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:51.405555    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:55:51.407601 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:51.407581    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 20:55:51.668595 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:51.668563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" event={"ID":"c262d8ce-d59b-4222-a950-101197b5a722","Type":"ContainerStarted","Data":"baac8485ad7cd1501e4556ae3437777c78c28b5187637617e4093d7e92c92797"}
Sep 18 20:55:51.668984 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:51.668625    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" event={"ID":"c262d8ce-d59b-4222-a950-101197b5a722","Type":"ContainerStarted","Data":"4eb7be87eac2cffb0599ea0d455d8256f7cfa717764e3f91db09ed82f13f3dce"}
Sep 18 20:55:51.668984 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:51.668827    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:51.689609 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:51.689551    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podStartSLOduration=2.321019676 podStartE2EDuration="25.689536153s" totalImagesPullingTime="19.489361578s" totalInitContainerRuntime="3.879154899s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:55:51.687857424 +0000 UTC m=+2102.595759696" watchObservedRunningTime="2026-09-18 20:55:51.689536153 +0000 UTC m=+2102.597438432"
Sep 18 20:55:52.673423 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:52.673390    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:52.674591 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:52.674562    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 20:55:53.677492 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:53.677444    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 20:55:58.682415 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:58.682384    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:55:58.682915 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:55:58.682887    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 20:56:00.427720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:00.427672    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:56:08.683328 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:08.683293    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 20:56:10.426930 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:10.426891    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:56:18.683214 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:18.683176    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 20:56:20.427730 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:20.427682    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:56:28.683544 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:28.683509    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 20:56:30.428347 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:30.428265    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:56:38.683645 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:38.683611    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 20:56:38.715308 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:38.715275    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"]
Sep 18 20:56:38.715536 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:38.715496    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container" containerID="cri-o://1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651" gracePeriod=30
Sep 18 20:56:38.715697 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:38.715536    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kube-rbac-proxy" containerID="cri-o://b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15" gracePeriod=30
Sep 18 20:56:38.895300 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:38.895269    2583 generic.go:356] "Generic (PLEG): container finished" podID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerID="b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15" exitCode=2
Sep 18 20:56:38.895443 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:38.895342    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" event={"ID":"8ce61be2-5593-41b3-8197-b45e37b11a02","Type":"ContainerDied","Data":"b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15"}
Sep 18 20:56:40.422886 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:40.422838    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 18 20:56:40.427483 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:40.427451    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:56:43.558869 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.558843    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:56:43.664503 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.664480    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ce61be2-5593-41b3-8197-b45e37b11a02-cabundle-cert\") pod \"8ce61be2-5593-41b3-8197-b45e37b11a02\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") "
Sep 18 20:56:43.664668 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.664539    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce61be2-5593-41b3-8197-b45e37b11a02-proxy-tls\") pod \"8ce61be2-5593-41b3-8197-b45e37b11a02\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") "
Sep 18 20:56:43.664668 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.664650    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce61be2-5593-41b3-8197-b45e37b11a02-kserve-provision-location\") pod \"8ce61be2-5593-41b3-8197-b45e37b11a02\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") "
Sep 18 20:56:43.664764 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.664716    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlgwz\" (UniqueName: \"kubernetes.io/projected/8ce61be2-5593-41b3-8197-b45e37b11a02-kube-api-access-rlgwz\") pod \"8ce61be2-5593-41b3-8197-b45e37b11a02\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") "
Sep 18 20:56:43.664764 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.664746    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce61be2-5593-41b3-8197-b45e37b11a02-kube-rbac-proxy-sar-config\") pod \"8ce61be2-5593-41b3-8197-b45e37b11a02\" (UID: \"8ce61be2-5593-41b3-8197-b45e37b11a02\") "
Sep 18 20:56:43.664978 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.664947    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce61be2-5593-41b3-8197-b45e37b11a02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ce61be2-5593-41b3-8197-b45e37b11a02" (UID: "8ce61be2-5593-41b3-8197-b45e37b11a02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:56:43.664978 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.664946    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce61be2-5593-41b3-8197-b45e37b11a02-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ce61be2-5593-41b3-8197-b45e37b11a02" (UID: "8ce61be2-5593-41b3-8197-b45e37b11a02"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:56:43.665247 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.665223    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce61be2-5593-41b3-8197-b45e37b11a02-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ce61be2-5593-41b3-8197-b45e37b11a02" (UID: "8ce61be2-5593-41b3-8197-b45e37b11a02"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:56:43.666927 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.666902    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce61be2-5593-41b3-8197-b45e37b11a02-kube-api-access-rlgwz" (OuterVolumeSpecName: "kube-api-access-rlgwz") pod "8ce61be2-5593-41b3-8197-b45e37b11a02" (UID: "8ce61be2-5593-41b3-8197-b45e37b11a02"). InnerVolumeSpecName "kube-api-access-rlgwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:56:43.666994 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.666934    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce61be2-5593-41b3-8197-b45e37b11a02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ce61be2-5593-41b3-8197-b45e37b11a02" (UID: "8ce61be2-5593-41b3-8197-b45e37b11a02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:56:43.765883 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.765851    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce61be2-5593-41b3-8197-b45e37b11a02-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:56:43.765883 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.765877    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce61be2-5593-41b3-8197-b45e37b11a02-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:56:43.765883 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.765888    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlgwz\" (UniqueName: \"kubernetes.io/projected/8ce61be2-5593-41b3-8197-b45e37b11a02-kube-api-access-rlgwz\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:56:43.766171 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.765899    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce61be2-5593-41b3-8197-b45e37b11a02-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:56:43.766171 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.765909    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ce61be2-5593-41b3-8197-b45e37b11a02-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:56:43.923382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.923280    2583 generic.go:356] "Generic (PLEG): container finished" podID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerID="1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651" exitCode=0
Sep 18 20:56:43.923382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.923357    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"
Sep 18 20:56:43.923382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.923362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" event={"ID":"8ce61be2-5593-41b3-8197-b45e37b11a02","Type":"ContainerDied","Data":"1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651"}
Sep 18 20:56:43.923682 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.923442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv" event={"ID":"8ce61be2-5593-41b3-8197-b45e37b11a02","Type":"ContainerDied","Data":"196148a05097101cf82e14246c60c785bef4d7b5be5c534ee87600ad6bc18681"}
Sep 18 20:56:43.923682 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.923474    2583 scope.go:117] "RemoveContainer" containerID="b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15"
Sep 18 20:56:43.933345 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.933328    2583 scope.go:117] "RemoveContainer" containerID="1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651"
Sep 18 20:56:43.941561 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.941543    2583 scope.go:117] "RemoveContainer" containerID="7586e41a8eb68a56aa91d7aa5523e681430dfa72648f265efbff9c037f242b8a"
Sep 18 20:56:43.949414 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.949391    2583 scope.go:117] "RemoveContainer" containerID="b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15"
Sep 18 20:56:43.949707 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:56:43.949682    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15\": container with ID starting with b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15 not found: ID does not exist" containerID="b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15"
Sep 18 20:56:43.949793 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.949719    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15"} err="failed to get container status \"b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15\": rpc error: code = NotFound desc = could not find container \"b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15\": container with ID starting with b1a86647d80ff91dde458f49292bc8cd9cb71d8ec1a4766b0c86ce55350f6e15 not found: ID does not exist"
Sep 18 20:56:43.949793 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.949741    2583 scope.go:117] "RemoveContainer" containerID="1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651"
Sep 18 20:56:43.950056 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:56:43.950034    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651\": container with ID starting with 1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651 not found: ID does not exist" containerID="1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651"
Sep 18 20:56:43.950102 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.950065    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651"} err="failed to get container status \"1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651\": rpc error: code = NotFound desc = could not find container \"1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651\": container with ID starting with 1aa396eef1b2561e1745df3161f8dbbcea606a6be3f370b2fdd5bbd33710b651 not found: ID does not exist"
Sep 18 20:56:43.950102 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.950082    2583 scope.go:117] "RemoveContainer" containerID="7586e41a8eb68a56aa91d7aa5523e681430dfa72648f265efbff9c037f242b8a"
Sep 18 20:56:43.950384 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:56:43.950364    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7586e41a8eb68a56aa91d7aa5523e681430dfa72648f265efbff9c037f242b8a\": container with ID starting with 7586e41a8eb68a56aa91d7aa5523e681430dfa72648f265efbff9c037f242b8a not found: ID does not exist" containerID="7586e41a8eb68a56aa91d7aa5523e681430dfa72648f265efbff9c037f242b8a"
Sep 18 20:56:43.950436 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.950390    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7586e41a8eb68a56aa91d7aa5523e681430dfa72648f265efbff9c037f242b8a"} err="failed to get container status \"7586e41a8eb68a56aa91d7aa5523e681430dfa72648f265efbff9c037f242b8a\": rpc error: code = NotFound desc = could not find container \"7586e41a8eb68a56aa91d7aa5523e681430dfa72648f265efbff9c037f242b8a\": container with ID starting with 7586e41a8eb68a56aa91d7aa5523e681430dfa72648f265efbff9c037f242b8a not found: ID does not exist"
Sep 18 20:56:43.950590 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.950574    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"]
Sep 18 20:56:43.956239 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.956215    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-88bbv"]
Sep 18 20:56:43.976151 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.976111    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"]
Sep 18 20:56:43.976477 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.976466    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container"
Sep 18 20:56:43.976535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.976479    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container"
Sep 18 20:56:43.976535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.976490    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kube-rbac-proxy"
Sep 18 20:56:43.976535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.976496    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kube-rbac-proxy"
Sep 18 20:56:43.976535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.976515    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="storage-initializer"
Sep 18 20:56:43.976535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.976521    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="storage-initializer"
Sep 18 20:56:43.976686 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.976569    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kube-rbac-proxy"
Sep 18 20:56:43.976686 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.976576    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" containerName="kserve-container"
Sep 18 20:56:43.978444 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.978428    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:43.980939 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.980919    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 20:56:43.981003 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.980976    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 20:56:43.989465 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:43.989447    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"]
Sep 18 20:56:44.068744 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.068698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7xzw\" (UniqueName: \"kubernetes.io/projected/daa98208-f4dc-4154-9297-a1c179833ff7-kube-api-access-b7xzw\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.068919 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.068766    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa98208-f4dc-4154-9297-a1c179833ff7-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.068919 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.068788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa98208-f4dc-4154-9297-a1c179833ff7-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.068919 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.068810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa98208-f4dc-4154-9297-a1c179833ff7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.068919 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.068828    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa98208-f4dc-4154-9297-a1c179833ff7-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.169985 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.169931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa98208-f4dc-4154-9297-a1c179833ff7-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.169985 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.169984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa98208-f4dc-4154-9297-a1c179833ff7-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.170281 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.170022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa98208-f4dc-4154-9297-a1c179833ff7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.170281 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.170047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa98208-f4dc-4154-9297-a1c179833ff7-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.170281 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.170156    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7xzw\" (UniqueName: \"kubernetes.io/projected/daa98208-f4dc-4154-9297-a1c179833ff7-kube-api-access-b7xzw\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.170559 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.170538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa98208-f4dc-4154-9297-a1c179833ff7-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.170749 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.170731    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa98208-f4dc-4154-9297-a1c179833ff7-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.170998 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.170976    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa98208-f4dc-4154-9297-a1c179833ff7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.172641 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.172626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa98208-f4dc-4154-9297-a1c179833ff7-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.179748 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.179701    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7xzw\" (UniqueName: \"kubernetes.io/projected/daa98208-f4dc-4154-9297-a1c179833ff7-kube-api-access-b7xzw\") pod \"isvc-canary-stop-predictor-5ffdccb94c-mpmnd\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.288635 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.288588    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:44.558887 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.558757    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"]
Sep 18 20:56:44.561384 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:56:44.561358    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa98208_f4dc_4154_9297_a1c179833ff7.slice/crio-c3f789cda556907f3a3e40c55cb0953ff2d9b6e9537e4fb5eadf7bf1590913fe WatchSource:0}: Error finding container c3f789cda556907f3a3e40c55cb0953ff2d9b6e9537e4fb5eadf7bf1590913fe: Status 404 returned error can't find the container with id c3f789cda556907f3a3e40c55cb0953ff2d9b6e9537e4fb5eadf7bf1590913fe
Sep 18 20:56:44.931080 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.931037    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" event={"ID":"daa98208-f4dc-4154-9297-a1c179833ff7","Type":"ContainerStarted","Data":"ef8ff2cbe41bbad39825e15988b9d3333e5463a483a80fb5e38bab7de5830e26"}
Sep 18 20:56:44.931287 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:44.931149    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" event={"ID":"daa98208-f4dc-4154-9297-a1c179833ff7","Type":"ContainerStarted","Data":"c3f789cda556907f3a3e40c55cb0953ff2d9b6e9537e4fb5eadf7bf1590913fe"}
Sep 18 20:56:45.316308 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:45.316230    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce61be2-5593-41b3-8197-b45e37b11a02" path="/var/lib/kubelet/pods/8ce61be2-5593-41b3-8197-b45e37b11a02/volumes"
Sep 18 20:56:48.682880 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:48.682844    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 20:56:48.951811 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:48.951780    2583 generic.go:356] "Generic (PLEG): container finished" podID="daa98208-f4dc-4154-9297-a1c179833ff7" containerID="ef8ff2cbe41bbad39825e15988b9d3333e5463a483a80fb5e38bab7de5830e26" exitCode=0
Sep 18 20:56:48.951953 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:48.951853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" event={"ID":"daa98208-f4dc-4154-9297-a1c179833ff7","Type":"ContainerDied","Data":"ef8ff2cbe41bbad39825e15988b9d3333e5463a483a80fb5e38bab7de5830e26"}
Sep 18 20:56:49.958086 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:49.958053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" event={"ID":"daa98208-f4dc-4154-9297-a1c179833ff7","Type":"ContainerStarted","Data":"db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443"}
Sep 18 20:56:49.958481 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:49.958137    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" event={"ID":"daa98208-f4dc-4154-9297-a1c179833ff7","Type":"ContainerStarted","Data":"04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef"}
Sep 18 20:56:49.958481 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:49.958348    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:49.985287 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:49.985241    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podStartSLOduration=3.127228754 podStartE2EDuration="6.985226679s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.857997925s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:56:49.983421727 +0000 UTC m=+2160.891324010" watchObservedRunningTime="2026-09-18 20:56:49.985226679 +0000 UTC m=+2160.893128998"
Sep 18 20:56:50.962742 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:50.962704    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:50.964067 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:50.964041    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:56:51.967697 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:51.967660    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:56:56.971934 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:56.971907    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:56:56.973389 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:56.972362    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:56:58.684341 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:56:58.684304    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:57:06.972588 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:06.972549    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:57:07.146055 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:07.146023    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"]
Sep 18 20:57:07.146310 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:07.146287    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container" containerID="cri-o://4eb7be87eac2cffb0599ea0d455d8256f7cfa717764e3f91db09ed82f13f3dce" gracePeriod=30
Sep 18 20:57:07.146412 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:07.146362    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kube-rbac-proxy" containerID="cri-o://baac8485ad7cd1501e4556ae3437777c78c28b5187637617e4093d7e92c92797" gracePeriod=30
Sep 18 20:57:08.046915 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:08.046885    2583 generic.go:356] "Generic (PLEG): container finished" podID="c262d8ce-d59b-4222-a950-101197b5a722" containerID="baac8485ad7cd1501e4556ae3437777c78c28b5187637617e4093d7e92c92797" exitCode=2
Sep 18 20:57:08.047388 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:08.046927    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" event={"ID":"c262d8ce-d59b-4222-a950-101197b5a722","Type":"ContainerDied","Data":"baac8485ad7cd1501e4556ae3437777c78c28b5187637617e4093d7e92c92797"}
Sep 18 20:57:08.677910 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:08.677872    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 18 20:57:08.683807 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:08.683767    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 20:57:11.063649 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.063617    2583 generic.go:356] "Generic (PLEG): container finished" podID="c262d8ce-d59b-4222-a950-101197b5a722" containerID="4eb7be87eac2cffb0599ea0d455d8256f7cfa717764e3f91db09ed82f13f3dce" exitCode=0
Sep 18 20:57:11.063984 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.063683    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" event={"ID":"c262d8ce-d59b-4222-a950-101197b5a722","Type":"ContainerDied","Data":"4eb7be87eac2cffb0599ea0d455d8256f7cfa717764e3f91db09ed82f13f3dce"}
Sep 18 20:57:11.093790 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.093768    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:57:11.210113 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.210032    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c262d8ce-d59b-4222-a950-101197b5a722-kube-rbac-proxy-sar-config\") pod \"c262d8ce-d59b-4222-a950-101197b5a722\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") "
Sep 18 20:57:11.210113 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.210068    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c262d8ce-d59b-4222-a950-101197b5a722-cabundle-cert\") pod \"c262d8ce-d59b-4222-a950-101197b5a722\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") "
Sep 18 20:57:11.210372 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.210115    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c262d8ce-d59b-4222-a950-101197b5a722-kserve-provision-location\") pod \"c262d8ce-d59b-4222-a950-101197b5a722\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") "
Sep 18 20:57:11.210372 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.210175    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbc44\" (UniqueName: \"kubernetes.io/projected/c262d8ce-d59b-4222-a950-101197b5a722-kube-api-access-dbc44\") pod \"c262d8ce-d59b-4222-a950-101197b5a722\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") "
Sep 18 20:57:11.210372 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.210250    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c262d8ce-d59b-4222-a950-101197b5a722-proxy-tls\") pod \"c262d8ce-d59b-4222-a950-101197b5a722\" (UID: \"c262d8ce-d59b-4222-a950-101197b5a722\") "
Sep 18 20:57:11.210535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.210478    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c262d8ce-d59b-4222-a950-101197b5a722-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c262d8ce-d59b-4222-a950-101197b5a722" (UID: "c262d8ce-d59b-4222-a950-101197b5a722"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:57:11.210535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.210481    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c262d8ce-d59b-4222-a950-101197b5a722-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c262d8ce-d59b-4222-a950-101197b5a722" (UID: "c262d8ce-d59b-4222-a950-101197b5a722"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:57:11.210535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.210485    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c262d8ce-d59b-4222-a950-101197b5a722-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c262d8ce-d59b-4222-a950-101197b5a722" (UID: "c262d8ce-d59b-4222-a950-101197b5a722"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:57:11.210641 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.210549    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c262d8ce-d59b-4222-a950-101197b5a722-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:57:11.210641 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.210567    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c262d8ce-d59b-4222-a950-101197b5a722-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:57:11.210641 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.210584    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c262d8ce-d59b-4222-a950-101197b5a722-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:57:11.212406 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.212382    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c262d8ce-d59b-4222-a950-101197b5a722-kube-api-access-dbc44" (OuterVolumeSpecName: "kube-api-access-dbc44") pod "c262d8ce-d59b-4222-a950-101197b5a722" (UID: "c262d8ce-d59b-4222-a950-101197b5a722"). InnerVolumeSpecName "kube-api-access-dbc44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:57:11.212512 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.212463    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c262d8ce-d59b-4222-a950-101197b5a722-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c262d8ce-d59b-4222-a950-101197b5a722" (UID: "c262d8ce-d59b-4222-a950-101197b5a722"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:57:11.311776 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.311748    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbc44\" (UniqueName: \"kubernetes.io/projected/c262d8ce-d59b-4222-a950-101197b5a722-kube-api-access-dbc44\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:57:11.311876 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:11.311778    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c262d8ce-d59b-4222-a950-101197b5a722-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:57:12.071022 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.070988    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw" event={"ID":"c262d8ce-d59b-4222-a950-101197b5a722","Type":"ContainerDied","Data":"ab776f6c7ac88284f9e747ff9742a0ae5bf7fc53724357ffe949a71e41d9a25f"}
Sep 18 20:57:12.071022 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.071004    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"
Sep 18 20:57:12.071535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.071053    2583 scope.go:117] "RemoveContainer" containerID="baac8485ad7cd1501e4556ae3437777c78c28b5187637617e4093d7e92c92797"
Sep 18 20:57:12.079526 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.079508    2583 scope.go:117] "RemoveContainer" containerID="4eb7be87eac2cffb0599ea0d455d8256f7cfa717764e3f91db09ed82f13f3dce"
Sep 18 20:57:12.087191 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.087175    2583 scope.go:117] "RemoveContainer" containerID="d8517e104268ae1b9fc443da9d943e0fa7f1b675484068d449814116c475918c"
Sep 18 20:57:12.090343 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.090323    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"]
Sep 18 20:57:12.094599 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.094579    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wvnhw"]
Sep 18 20:57:12.368019 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.367943    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"]
Sep 18 20:57:12.368334 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.368319    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="storage-initializer"
Sep 18 20:57:12.368382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.368336    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="storage-initializer"
Sep 18 20:57:12.368382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.368351    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container"
Sep 18 20:57:12.368382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.368356    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container"
Sep 18 20:57:12.368382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.368374    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kube-rbac-proxy"
Sep 18 20:57:12.368382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.368380    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kube-rbac-proxy"
Sep 18 20:57:12.368537 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.368464    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kserve-container"
Sep 18 20:57:12.368537 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.368471    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c262d8ce-d59b-4222-a950-101197b5a722" containerName="kube-rbac-proxy"
Sep 18 20:57:12.371541 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.371520    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.383111 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.376679    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 18 20:57:12.383111 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.377094    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 18 20:57:12.385169 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.385115    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"]
Sep 18 20:57:12.419115 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.419085    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.419115 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.419138    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.419306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.419188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.419306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.419242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.419306 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.419284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7wh\" (UniqueName: \"kubernetes.io/projected/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kube-api-access-wl7wh\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.520094 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.520061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.520270 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.520111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.520270 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.520154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.520270 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.520190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.520270 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.520217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7wh\" (UniqueName: \"kubernetes.io/projected/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kube-api-access-wl7wh\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.520569 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.520545    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.520826 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.520806    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.520924 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.520901    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.522789 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.522771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.529050 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.529027    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7wh\" (UniqueName: \"kubernetes.io/projected/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kube-api-access-wl7wh\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.684737 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.684706    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:12.956843 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:12.956724    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"]
Sep 18 20:57:12.959200 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:57:12.959172    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b1cb36_1c38_4a79_8f5b_00c4adcd25bb.slice/crio-4d3cd7c18ca1ee2261b8e464cae0abcb65b00582f536239a6ecc809ccc49f30d WatchSource:0}: Error finding container 4d3cd7c18ca1ee2261b8e464cae0abcb65b00582f536239a6ecc809ccc49f30d: Status 404 returned error can't find the container with id 4d3cd7c18ca1ee2261b8e464cae0abcb65b00582f536239a6ecc809ccc49f30d
Sep 18 20:57:13.077201 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:13.077148    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd" event={"ID":"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb","Type":"ContainerStarted","Data":"79184093103a68b231b77a1e333c21e7c621c4c983a531e734beb0d4905e872a"}
Sep 18 20:57:13.077636 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:13.077227    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd" event={"ID":"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb","Type":"ContainerStarted","Data":"4d3cd7c18ca1ee2261b8e464cae0abcb65b00582f536239a6ecc809ccc49f30d"}
Sep 18 20:57:13.316168 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:13.316067    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c262d8ce-d59b-4222-a950-101197b5a722" path="/var/lib/kubelet/pods/c262d8ce-d59b-4222-a950-101197b5a722/volumes"
Sep 18 20:57:16.972809 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:16.972775    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:57:17.101253 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:17.101216    2583 generic.go:356] "Generic (PLEG): container finished" podID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerID="79184093103a68b231b77a1e333c21e7c621c4c983a531e734beb0d4905e872a" exitCode=0
Sep 18 20:57:17.101409 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:17.101288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd" event={"ID":"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb","Type":"ContainerDied","Data":"79184093103a68b231b77a1e333c21e7c621c4c983a531e734beb0d4905e872a"}
Sep 18 20:57:18.107996 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:18.107961    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd" event={"ID":"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb","Type":"ContainerStarted","Data":"104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56"}
Sep 18 20:57:18.108477 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:18.108043    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd" event={"ID":"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb","Type":"ContainerStarted","Data":"2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d"}
Sep 18 20:57:18.108477 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:18.108236    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:18.108477 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:18.108291    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:18.130710 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:18.130654    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd" podStartSLOduration=2.181959899 podStartE2EDuration="6.130638903s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.948679004s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:57:18.127782421 +0000 UTC m=+2189.035684707" watchObservedRunningTime="2026-09-18 20:57:18.130638903 +0000 UTC m=+2189.038541182"
Sep 18 20:57:24.119026 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:24.118997    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:26.972429 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:26.972391    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:57:36.972944 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:36.972900    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:57:46.973285 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:46.973241    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:57:54.123851 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:54.123815    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:57:56.973302 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:57:56.973259    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:58:02.424267 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:02.424172    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"]
Sep 18 20:58:02.424726 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:02.424501    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd" podUID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerName="kserve-container" containerID="cri-o://2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d" gracePeriod=30
Sep 18 20:58:02.424726 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:02.424540    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd" podUID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerName="kube-rbac-proxy" containerID="cri-o://104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56" gracePeriod=30
Sep 18 20:58:03.316179 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:03.316141    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:58:03.328290 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:03.328262    2583 generic.go:356] "Generic (PLEG): container finished" podID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerID="104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56" exitCode=2
Sep 18 20:58:03.328467 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:03.328344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd" event={"ID":"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb","Type":"ContainerDied","Data":"104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56"}
Sep 18 20:58:04.114426 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:04.114384    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd" podUID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 18 20:58:07.624501 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.624465    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"]
Sep 18 20:58:07.628139 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.628099    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.630586 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.630562    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 18 20:58:07.630761 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.630721    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 18 20:58:07.638980 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.638954    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"]
Sep 18 20:58:07.699258 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.699229    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68f93fa1-4112-43e7-8f84-685c5d50cf61-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.699412 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.699288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68f93fa1-4112-43e7-8f84-685c5d50cf61-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.699412 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.699348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68f93fa1-4112-43e7-8f84-685c5d50cf61-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.699412 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.699409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68f93fa1-4112-43e7-8f84-685c5d50cf61-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.699528 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.699430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc55b\" (UniqueName: \"kubernetes.io/projected/68f93fa1-4112-43e7-8f84-685c5d50cf61-kube-api-access-lc55b\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.800859 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.800825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68f93fa1-4112-43e7-8f84-685c5d50cf61-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.801038 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.800877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68f93fa1-4112-43e7-8f84-685c5d50cf61-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.801082 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.801041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68f93fa1-4112-43e7-8f84-685c5d50cf61-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.801142 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.801101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68f93fa1-4112-43e7-8f84-685c5d50cf61-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.801276 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.801147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc55b\" (UniqueName: \"kubernetes.io/projected/68f93fa1-4112-43e7-8f84-685c5d50cf61-kube-api-access-lc55b\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.801276 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.801242    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68f93fa1-4112-43e7-8f84-685c5d50cf61-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.801699 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.801676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68f93fa1-4112-43e7-8f84-685c5d50cf61-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.801788 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.801710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68f93fa1-4112-43e7-8f84-685c5d50cf61-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.803657 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.803637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68f93fa1-4112-43e7-8f84-685c5d50cf61-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.809059 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.809037    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc55b\" (UniqueName: \"kubernetes.io/projected/68f93fa1-4112-43e7-8f84-685c5d50cf61-kube-api-access-lc55b\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:07.941146 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:07.941085    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:08.214135 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:08.214090    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"]
Sep 18 20:58:08.216797 ip-10-0-130-79 kubenswrapper[2583]: W0918 20:58:08.216757    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f93fa1_4112_43e7_8f84_685c5d50cf61.slice/crio-32df5cc49716220eae8ae7e2f08befe29b18fec5e72c529dc205b721b3cd75fd WatchSource:0}: Error finding container 32df5cc49716220eae8ae7e2f08befe29b18fec5e72c529dc205b721b3cd75fd: Status 404 returned error can't find the container with id 32df5cc49716220eae8ae7e2f08befe29b18fec5e72c529dc205b721b3cd75fd
Sep 18 20:58:08.351649 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:08.351616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" event={"ID":"68f93fa1-4112-43e7-8f84-685c5d50cf61","Type":"ContainerStarted","Data":"661e825080c17ea78af7e607cfed03d62b6d7b9f6307d1f9c0e5c91eb3d5ae1b"}
Sep 18 20:58:08.351819 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:08.351678    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" event={"ID":"68f93fa1-4112-43e7-8f84-685c5d50cf61","Type":"ContainerStarted","Data":"32df5cc49716220eae8ae7e2f08befe29b18fec5e72c529dc205b721b3cd75fd"}
Sep 18 20:58:08.903547 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:08.903521    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:58:09.013064 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.012994    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-cabundle-cert\") pod \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") "
Sep 18 20:58:09.013064 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.013023    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-proxy-tls\") pod \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") "
Sep 18 20:58:09.013307 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.013070    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kserve-provision-location\") pod \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") "
Sep 18 20:58:09.013307 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.013158    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kube-rbac-proxy-sar-config\") pod \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") "
Sep 18 20:58:09.013307 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.013196    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl7wh\" (UniqueName: \"kubernetes.io/projected/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kube-api-access-wl7wh\") pod \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\" (UID: \"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb\") "
Sep 18 20:58:09.013468 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.013352    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" (UID: "e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:58:09.013528 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.013503    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" (UID: "e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:58:09.013528 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.013517    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:58:09.013601 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.013548    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" (UID: "e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:58:09.015301 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.015286    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" (UID: "e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:58:09.015395 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.015382    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kube-api-access-wl7wh" (OuterVolumeSpecName: "kube-api-access-wl7wh") pod "e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" (UID: "e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb"). InnerVolumeSpecName "kube-api-access-wl7wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:58:09.114087 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.114064    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:58:09.114087 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.114083    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl7wh\" (UniqueName: \"kubernetes.io/projected/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kube-api-access-wl7wh\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:58:09.114252 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.114092    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:58:09.114252 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.114102    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:58:09.358143 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.358103    2583 generic.go:356] "Generic (PLEG): container finished" podID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerID="2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d" exitCode=0
Sep 18 20:58:09.358281 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.358155    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd" event={"ID":"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb","Type":"ContainerDied","Data":"2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d"}
Sep 18 20:58:09.358281 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.358182    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"
Sep 18 20:58:09.358281 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.358213    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd" event={"ID":"e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb","Type":"ContainerDied","Data":"4d3cd7c18ca1ee2261b8e464cae0abcb65b00582f536239a6ecc809ccc49f30d"}
Sep 18 20:58:09.358281 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.358229    2583 scope.go:117] "RemoveContainer" containerID="104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56"
Sep 18 20:58:09.369410 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.369395    2583 scope.go:117] "RemoveContainer" containerID="2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d"
Sep 18 20:58:09.377870 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.377854    2583 scope.go:117] "RemoveContainer" containerID="79184093103a68b231b77a1e333c21e7c621c4c983a531e734beb0d4905e872a"
Sep 18 20:58:09.378921 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.378904    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"]
Sep 18 20:58:09.383142 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.383097    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hddhd"]
Sep 18 20:58:09.386649 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.386634    2583 scope.go:117] "RemoveContainer" containerID="104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56"
Sep 18 20:58:09.386929 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:58:09.386913    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56\": container with ID starting with 104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56 not found: ID does not exist" containerID="104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56"
Sep 18 20:58:09.386976 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.386939    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56"} err="failed to get container status \"104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56\": rpc error: code = NotFound desc = could not find container \"104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56\": container with ID starting with 104d8dd77d0e8eb87bd7a55dae3e7b22f604738227d9ebead915827169fbfd56 not found: ID does not exist"
Sep 18 20:58:09.386976 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.386955    2583 scope.go:117] "RemoveContainer" containerID="2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d"
Sep 18 20:58:09.387200 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:58:09.387185    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d\": container with ID starting with 2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d not found: ID does not exist" containerID="2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d"
Sep 18 20:58:09.387246 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.387206    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d"} err="failed to get container status \"2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d\": rpc error: code = NotFound desc = could not find container \"2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d\": container with ID starting with 2fe84058badd485e8b3e03bc8930377523505a29290a5fd9b3a0fb8d84172e6d not found: ID does not exist"
Sep 18 20:58:09.387246 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.387219    2583 scope.go:117] "RemoveContainer" containerID="79184093103a68b231b77a1e333c21e7c621c4c983a531e734beb0d4905e872a"
Sep 18 20:58:09.387446 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:58:09.387432    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79184093103a68b231b77a1e333c21e7c621c4c983a531e734beb0d4905e872a\": container with ID starting with 79184093103a68b231b77a1e333c21e7c621c4c983a531e734beb0d4905e872a not found: ID does not exist" containerID="79184093103a68b231b77a1e333c21e7c621c4c983a531e734beb0d4905e872a"
Sep 18 20:58:09.387486 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:09.387450    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79184093103a68b231b77a1e333c21e7c621c4c983a531e734beb0d4905e872a"} err="failed to get container status \"79184093103a68b231b77a1e333c21e7c621c4c983a531e734beb0d4905e872a\": rpc error: code = NotFound desc = could not find container \"79184093103a68b231b77a1e333c21e7c621c4c983a531e734beb0d4905e872a\": container with ID starting with 79184093103a68b231b77a1e333c21e7c621c4c983a531e734beb0d4905e872a not found: ID does not exist"
Sep 18 20:58:11.316603 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:11.316568    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" path="/var/lib/kubelet/pods/e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb/volumes"
Sep 18 20:58:12.374774 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:12.374698    2583 generic.go:356] "Generic (PLEG): container finished" podID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerID="661e825080c17ea78af7e607cfed03d62b6d7b9f6307d1f9c0e5c91eb3d5ae1b" exitCode=0
Sep 18 20:58:12.375224 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:12.374768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" event={"ID":"68f93fa1-4112-43e7-8f84-685c5d50cf61","Type":"ContainerDied","Data":"661e825080c17ea78af7e607cfed03d62b6d7b9f6307d1f9c0e5c91eb3d5ae1b"}
Sep 18 20:58:13.317878 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:13.317849    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:58:13.382767 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:13.382738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" event={"ID":"68f93fa1-4112-43e7-8f84-685c5d50cf61","Type":"ContainerStarted","Data":"25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e"}
Sep 18 20:58:13.383255 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:13.382809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" event={"ID":"68f93fa1-4112-43e7-8f84-685c5d50cf61","Type":"ContainerStarted","Data":"4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371"}
Sep 18 20:58:13.383255 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:13.383033    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:13.403272 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:13.403226    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" podStartSLOduration=2.613563868 podStartE2EDuration="6.403209865s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.789645997s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:58:13.401523439 +0000 UTC m=+2244.309425731" watchObservedRunningTime="2026-09-18 20:58:13.403209865 +0000 UTC m=+2244.311112143"
Sep 18 20:58:14.387903 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:14.387871    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:14.573741 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:14.573710    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"]
Sep 18 20:58:14.573929 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:14.573910    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container" containerID="cri-o://04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef" gracePeriod=30
Sep 18 20:58:14.574006 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:14.573982    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kube-rbac-proxy" containerID="cri-o://db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443" gracePeriod=30
Sep 18 20:58:15.393534 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:15.393510    2583 generic.go:356] "Generic (PLEG): container finished" podID="daa98208-f4dc-4154-9297-a1c179833ff7" containerID="db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443" exitCode=2
Sep 18 20:58:15.393856 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:15.393581    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" event={"ID":"daa98208-f4dc-4154-9297-a1c179833ff7","Type":"ContainerDied","Data":"db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443"}
Sep 18 20:58:16.968632 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:16.968593    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 18 20:58:19.327854 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.327824    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:58:19.414664 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.414633    2583 generic.go:356] "Generic (PLEG): container finished" podID="daa98208-f4dc-4154-9297-a1c179833ff7" containerID="04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef" exitCode=0
Sep 18 20:58:19.414840 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.414695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" event={"ID":"daa98208-f4dc-4154-9297-a1c179833ff7","Type":"ContainerDied","Data":"04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef"}
Sep 18 20:58:19.414840 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.414744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd" event={"ID":"daa98208-f4dc-4154-9297-a1c179833ff7","Type":"ContainerDied","Data":"c3f789cda556907f3a3e40c55cb0953ff2d9b6e9537e4fb5eadf7bf1590913fe"}
Sep 18 20:58:19.414840 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.414768    2583 scope.go:117] "RemoveContainer" containerID="db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443"
Sep 18 20:58:19.414840 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.414697    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"
Sep 18 20:58:19.423172 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.423155    2583 scope.go:117] "RemoveContainer" containerID="04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef"
Sep 18 20:58:19.431598 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.431580    2583 scope.go:117] "RemoveContainer" containerID="ef8ff2cbe41bbad39825e15988b9d3333e5463a483a80fb5e38bab7de5830e26"
Sep 18 20:58:19.438996 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.438981    2583 scope.go:117] "RemoveContainer" containerID="db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443"
Sep 18 20:58:19.439302 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:58:19.439281    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443\": container with ID starting with db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443 not found: ID does not exist" containerID="db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443"
Sep 18 20:58:19.439367 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.439311    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443"} err="failed to get container status \"db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443\": rpc error: code = NotFound desc = could not find container \"db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443\": container with ID starting with db88c4c5b477b6295a1eb8e6a611479cc410718466f0022116fa94784aee5443 not found: ID does not exist"
Sep 18 20:58:19.439367 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.439328    2583 scope.go:117] "RemoveContainer" containerID="04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef"
Sep 18 20:58:19.439557 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:58:19.439543    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef\": container with ID starting with 04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef not found: ID does not exist" containerID="04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef"
Sep 18 20:58:19.439609 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.439560    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef"} err="failed to get container status \"04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef\": rpc error: code = NotFound desc = could not find container \"04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef\": container with ID starting with 04993ed38c6679ddcb607fcb9e058e6d07ead599d6bd9e6c4179fea3aa0c08ef not found: ID does not exist"
Sep 18 20:58:19.439609 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.439571    2583 scope.go:117] "RemoveContainer" containerID="ef8ff2cbe41bbad39825e15988b9d3333e5463a483a80fb5e38bab7de5830e26"
Sep 18 20:58:19.439800 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:58:19.439778    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8ff2cbe41bbad39825e15988b9d3333e5463a483a80fb5e38bab7de5830e26\": container with ID starting with ef8ff2cbe41bbad39825e15988b9d3333e5463a483a80fb5e38bab7de5830e26 not found: ID does not exist" containerID="ef8ff2cbe41bbad39825e15988b9d3333e5463a483a80fb5e38bab7de5830e26"
Sep 18 20:58:19.439837 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.439801    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8ff2cbe41bbad39825e15988b9d3333e5463a483a80fb5e38bab7de5830e26"} err="failed to get container status \"ef8ff2cbe41bbad39825e15988b9d3333e5463a483a80fb5e38bab7de5830e26\": rpc error: code = NotFound desc = could not find container \"ef8ff2cbe41bbad39825e15988b9d3333e5463a483a80fb5e38bab7de5830e26\": container with ID starting with ef8ff2cbe41bbad39825e15988b9d3333e5463a483a80fb5e38bab7de5830e26 not found: ID does not exist"
Sep 18 20:58:19.503136 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.503095    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa98208-f4dc-4154-9297-a1c179833ff7-cabundle-cert\") pod \"daa98208-f4dc-4154-9297-a1c179833ff7\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") "
Sep 18 20:58:19.503257 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.503197    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa98208-f4dc-4154-9297-a1c179833ff7-kserve-provision-location\") pod \"daa98208-f4dc-4154-9297-a1c179833ff7\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") "
Sep 18 20:58:19.503257 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.503235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7xzw\" (UniqueName: \"kubernetes.io/projected/daa98208-f4dc-4154-9297-a1c179833ff7-kube-api-access-b7xzw\") pod \"daa98208-f4dc-4154-9297-a1c179833ff7\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") "
Sep 18 20:58:19.503342 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.503259    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa98208-f4dc-4154-9297-a1c179833ff7-proxy-tls\") pod \"daa98208-f4dc-4154-9297-a1c179833ff7\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") "
Sep 18 20:58:19.503342 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.503301    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa98208-f4dc-4154-9297-a1c179833ff7-kube-rbac-proxy-sar-config\") pod \"daa98208-f4dc-4154-9297-a1c179833ff7\" (UID: \"daa98208-f4dc-4154-9297-a1c179833ff7\") "
Sep 18 20:58:19.503517 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.503488    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa98208-f4dc-4154-9297-a1c179833ff7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "daa98208-f4dc-4154-9297-a1c179833ff7" (UID: "daa98208-f4dc-4154-9297-a1c179833ff7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:58:19.503588 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.503494    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa98208-f4dc-4154-9297-a1c179833ff7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "daa98208-f4dc-4154-9297-a1c179833ff7" (UID: "daa98208-f4dc-4154-9297-a1c179833ff7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:58:19.503720 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.503697    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa98208-f4dc-4154-9297-a1c179833ff7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "daa98208-f4dc-4154-9297-a1c179833ff7" (UID: "daa98208-f4dc-4154-9297-a1c179833ff7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:58:19.505476 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.505456    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa98208-f4dc-4154-9297-a1c179833ff7-kube-api-access-b7xzw" (OuterVolumeSpecName: "kube-api-access-b7xzw") pod "daa98208-f4dc-4154-9297-a1c179833ff7" (UID: "daa98208-f4dc-4154-9297-a1c179833ff7"). InnerVolumeSpecName "kube-api-access-b7xzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:58:19.505535 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.505471    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa98208-f4dc-4154-9297-a1c179833ff7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "daa98208-f4dc-4154-9297-a1c179833ff7" (UID: "daa98208-f4dc-4154-9297-a1c179833ff7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:58:19.604219 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.604195    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa98208-f4dc-4154-9297-a1c179833ff7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:58:19.604219 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.604220    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa98208-f4dc-4154-9297-a1c179833ff7-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:58:19.604359 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.604230    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa98208-f4dc-4154-9297-a1c179833ff7-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:58:19.604359 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.604240    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7xzw\" (UniqueName: \"kubernetes.io/projected/daa98208-f4dc-4154-9297-a1c179833ff7-kube-api-access-b7xzw\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:58:19.604359 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.604249    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa98208-f4dc-4154-9297-a1c179833ff7-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:58:19.738154 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.738109    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"]
Sep 18 20:58:19.742671 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:19.742648    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-mpmnd"]
Sep 18 20:58:20.399666 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:20.399637    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:21.316069 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:21.316039    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" path="/var/lib/kubelet/pods/daa98208-f4dc-4154-9297-a1c179833ff7/volumes"
Sep 18 20:58:50.403768 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:50.403738    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:58:57.704575 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:57.704531    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"]
Sep 18 20:58:57.705021 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:57.704830    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerName="kserve-container" containerID="cri-o://4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371" gracePeriod=30
Sep 18 20:58:57.705021 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:57.704855    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerName="kube-rbac-proxy" containerID="cri-o://25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e" gracePeriod=30
Sep 18 20:58:58.594750 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:58.594711    2583 generic.go:356] "Generic (PLEG): container finished" podID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerID="25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e" exitCode=2
Sep 18 20:58:58.594921 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:58:58.594765    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" event={"ID":"68f93fa1-4112-43e7-8f84-685c5d50cf61","Type":"ContainerDied","Data":"25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e"}
Sep 18 20:59:00.394319 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:00.394272    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 18 20:59:00.400205 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:00.400176    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 20:59:03.959304 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:03.959275    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:59:04.077888 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.077804    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68f93fa1-4112-43e7-8f84-685c5d50cf61-kube-rbac-proxy-sar-config\") pod \"68f93fa1-4112-43e7-8f84-685c5d50cf61\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") "
Sep 18 20:59:04.077888 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.077838    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc55b\" (UniqueName: \"kubernetes.io/projected/68f93fa1-4112-43e7-8f84-685c5d50cf61-kube-api-access-lc55b\") pod \"68f93fa1-4112-43e7-8f84-685c5d50cf61\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") "
Sep 18 20:59:04.077888 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.077869    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68f93fa1-4112-43e7-8f84-685c5d50cf61-cabundle-cert\") pod \"68f93fa1-4112-43e7-8f84-685c5d50cf61\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") "
Sep 18 20:59:04.078193 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.077918    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68f93fa1-4112-43e7-8f84-685c5d50cf61-proxy-tls\") pod \"68f93fa1-4112-43e7-8f84-685c5d50cf61\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") "
Sep 18 20:59:04.078193 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.077938    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68f93fa1-4112-43e7-8f84-685c5d50cf61-kserve-provision-location\") pod \"68f93fa1-4112-43e7-8f84-685c5d50cf61\" (UID: \"68f93fa1-4112-43e7-8f84-685c5d50cf61\") "
Sep 18 20:59:04.078329 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.078301    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f93fa1-4112-43e7-8f84-685c5d50cf61-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "68f93fa1-4112-43e7-8f84-685c5d50cf61" (UID: "68f93fa1-4112-43e7-8f84-685c5d50cf61"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:59:04.078390 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.078363    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f93fa1-4112-43e7-8f84-685c5d50cf61-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "68f93fa1-4112-43e7-8f84-685c5d50cf61" (UID: "68f93fa1-4112-43e7-8f84-685c5d50cf61"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:59:04.078438 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.078395    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f93fa1-4112-43e7-8f84-685c5d50cf61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68f93fa1-4112-43e7-8f84-685c5d50cf61" (UID: "68f93fa1-4112-43e7-8f84-685c5d50cf61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:59:04.080382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.080349    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f93fa1-4112-43e7-8f84-685c5d50cf61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68f93fa1-4112-43e7-8f84-685c5d50cf61" (UID: "68f93fa1-4112-43e7-8f84-685c5d50cf61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:59:04.080382 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.080359    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f93fa1-4112-43e7-8f84-685c5d50cf61-kube-api-access-lc55b" (OuterVolumeSpecName: "kube-api-access-lc55b") pod "68f93fa1-4112-43e7-8f84-685c5d50cf61" (UID: "68f93fa1-4112-43e7-8f84-685c5d50cf61"). InnerVolumeSpecName "kube-api-access-lc55b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:59:04.179090 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.179056    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68f93fa1-4112-43e7-8f84-685c5d50cf61-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:59:04.179090 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.179090    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc55b\" (UniqueName: \"kubernetes.io/projected/68f93fa1-4112-43e7-8f84-685c5d50cf61-kube-api-access-lc55b\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:59:04.179284 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.179106    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68f93fa1-4112-43e7-8f84-685c5d50cf61-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:59:04.179284 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.179149    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68f93fa1-4112-43e7-8f84-685c5d50cf61-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:59:04.179284 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.179163    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68f93fa1-4112-43e7-8f84-685c5d50cf61-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 20:59:04.623195 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.623164    2583 generic.go:356] "Generic (PLEG): container finished" podID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerID="4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371" exitCode=0
Sep 18 20:59:04.623365 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.623211    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" event={"ID":"68f93fa1-4112-43e7-8f84-685c5d50cf61","Type":"ContainerDied","Data":"4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371"}
Sep 18 20:59:04.623365 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.623225    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"
Sep 18 20:59:04.623365 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.623261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv" event={"ID":"68f93fa1-4112-43e7-8f84-685c5d50cf61","Type":"ContainerDied","Data":"32df5cc49716220eae8ae7e2f08befe29b18fec5e72c529dc205b721b3cd75fd"}
Sep 18 20:59:04.623365 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.623287    2583 scope.go:117] "RemoveContainer" containerID="25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e"
Sep 18 20:59:04.632260 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.632243    2583 scope.go:117] "RemoveContainer" containerID="4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371"
Sep 18 20:59:04.639833 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.639816    2583 scope.go:117] "RemoveContainer" containerID="661e825080c17ea78af7e607cfed03d62b6d7b9f6307d1f9c0e5c91eb3d5ae1b"
Sep 18 20:59:04.644270 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.644249    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"]
Sep 18 20:59:04.648315 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.648298    2583 scope.go:117] "RemoveContainer" containerID="25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e"
Sep 18 20:59:04.648626 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.648607    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9vlpv"]
Sep 18 20:59:04.648683 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:59:04.648630    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e\": container with ID starting with 25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e not found: ID does not exist" containerID="25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e"
Sep 18 20:59:04.648683 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.648657    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e"} err="failed to get container status \"25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e\": rpc error: code = NotFound desc = could not find container \"25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e\": container with ID starting with 25f976997d223698b578e3c79a9a6fbe0b3f88e8b133739d83b66f0360ab563e not found: ID does not exist"
Sep 18 20:59:04.648683 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.648674    2583 scope.go:117] "RemoveContainer" containerID="4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371"
Sep 18 20:59:04.648983 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:59:04.648965    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371\": container with ID starting with 4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371 not found: ID does not exist" containerID="4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371"
Sep 18 20:59:04.649023 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.648988    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371"} err="failed to get container status \"4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371\": rpc error: code = NotFound desc = could not find container \"4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371\": container with ID starting with 4e75ea4b7b53dc5e2eb6a862b120af6fc4057318eba032fc9c941c5f7606f371 not found: ID does not exist"
Sep 18 20:59:04.649023 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.649005    2583 scope.go:117] "RemoveContainer" containerID="661e825080c17ea78af7e607cfed03d62b6d7b9f6307d1f9c0e5c91eb3d5ae1b"
Sep 18 20:59:04.649295 ip-10-0-130-79 kubenswrapper[2583]: E0918 20:59:04.649277    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661e825080c17ea78af7e607cfed03d62b6d7b9f6307d1f9c0e5c91eb3d5ae1b\": container with ID starting with 661e825080c17ea78af7e607cfed03d62b6d7b9f6307d1f9c0e5c91eb3d5ae1b not found: ID does not exist" containerID="661e825080c17ea78af7e607cfed03d62b6d7b9f6307d1f9c0e5c91eb3d5ae1b"
Sep 18 20:59:04.649369 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:04.649302    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661e825080c17ea78af7e607cfed03d62b6d7b9f6307d1f9c0e5c91eb3d5ae1b"} err="failed to get container status \"661e825080c17ea78af7e607cfed03d62b6d7b9f6307d1f9c0e5c91eb3d5ae1b\": rpc error: code = NotFound desc = could not find container \"661e825080c17ea78af7e607cfed03d62b6d7b9f6307d1f9c0e5c91eb3d5ae1b\": container with ID starting with 661e825080c17ea78af7e607cfed03d62b6d7b9f6307d1f9c0e5c91eb3d5ae1b not found: ID does not exist"
Sep 18 20:59:05.317637 ip-10-0-130-79 kubenswrapper[2583]: I0918 20:59:05.317597    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" path="/var/lib/kubelet/pods/68f93fa1-4112-43e7-8f84-685c5d50cf61/volumes"
Sep 18 21:00:48.773137 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773084    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"]
Sep 18 21:00:48.773554 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773476    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="storage-initializer"
Sep 18 21:00:48.773554 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773494    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="storage-initializer"
Sep 18 21:00:48.773554 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773511    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerName="storage-initializer"
Sep 18 21:00:48.773554 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773520    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerName="storage-initializer"
Sep 18 21:00:48.773554 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773529    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerName="storage-initializer"
Sep 18 21:00:48.773554 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773537    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerName="storage-initializer"
Sep 18 21:00:48.773554 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773553    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerName="kube-rbac-proxy"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773561    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerName="kube-rbac-proxy"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773581    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerName="kserve-container"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773590    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerName="kserve-container"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773599    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerName="kserve-container"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773604    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerName="kserve-container"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773612    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kube-rbac-proxy"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773617    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kube-rbac-proxy"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773624    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773630    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773636    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerName="kube-rbac-proxy"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773641    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerName="kube-rbac-proxy"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773708    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerName="kube-rbac-proxy"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773716    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kube-rbac-proxy"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773722    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="daa98208-f4dc-4154-9297-a1c179833ff7" containerName="kserve-container"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773728    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerName="kserve-container"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773734    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b1cb36-1c38-4a79-8f5b-00c4adcd25bb" containerName="kserve-container"
Sep 18 21:00:48.773778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.773740    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f93fa1-4112-43e7-8f84-685c5d50cf61" containerName="kube-rbac-proxy"
Sep 18 21:00:48.776831 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.776815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:48.779674 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.779645    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 21:00:48.779808 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.779699    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 21:00:48.779808 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.779699    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 21:00:48.780795 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.780709    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 18 21:00:48.780795 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.780715    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 21:00:48.780986 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.780969    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 21:00:48.781262 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.781219    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 21:00:48.787648 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.787628    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"]
Sep 18 21:00:48.939729 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.939699    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b5605fc-55bf-4ef0-b90e-7da5324321af-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:48.939729 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.939733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5605fc-55bf-4ef0-b90e-7da5324321af-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:48.939921 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.939801    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b5605fc-55bf-4ef0-b90e-7da5324321af-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:48.939921 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.939849    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv2bw\" (UniqueName: \"kubernetes.io/projected/7b5605fc-55bf-4ef0-b90e-7da5324321af-kube-api-access-qv2bw\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:48.939921 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:48.939866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b5605fc-55bf-4ef0-b90e-7da5324321af-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:49.041225 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:49.041154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b5605fc-55bf-4ef0-b90e-7da5324321af-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:49.041225 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:49.041198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv2bw\" (UniqueName: \"kubernetes.io/projected/7b5605fc-55bf-4ef0-b90e-7da5324321af-kube-api-access-qv2bw\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:49.041225 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:49.041215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b5605fc-55bf-4ef0-b90e-7da5324321af-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:49.041438 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:49.041250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b5605fc-55bf-4ef0-b90e-7da5324321af-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:49.041438 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:49.041268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5605fc-55bf-4ef0-b90e-7da5324321af-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:49.041613 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:49.041595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5605fc-55bf-4ef0-b90e-7da5324321af-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:49.041945 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:49.041922    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b5605fc-55bf-4ef0-b90e-7da5324321af-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:49.042027 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:49.041975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b5605fc-55bf-4ef0-b90e-7da5324321af-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:49.043740 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:49.043722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b5605fc-55bf-4ef0-b90e-7da5324321af-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:49.049470 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:49.049449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv2bw\" (UniqueName: \"kubernetes.io/projected/7b5605fc-55bf-4ef0-b90e-7da5324321af-kube-api-access-qv2bw\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:49.087843 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:49.087821    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:49.359010 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:49.358986    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"]
Sep 18 21:00:49.361207 ip-10-0-130-79 kubenswrapper[2583]: W0918 21:00:49.361182    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5605fc_55bf_4ef0_b90e_7da5324321af.slice/crio-bf8c9f47543b4ee128d1c579c75d44fc362d3e69dce445ecbcbd776dfad4272d WatchSource:0}: Error finding container bf8c9f47543b4ee128d1c579c75d44fc362d3e69dce445ecbcbd776dfad4272d: Status 404 returned error can't find the container with id bf8c9f47543b4ee128d1c579c75d44fc362d3e69dce445ecbcbd776dfad4272d
Sep 18 21:00:50.098283 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:50.098249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" event={"ID":"7b5605fc-55bf-4ef0-b90e-7da5324321af","Type":"ContainerStarted","Data":"e31fcee0b80fd0982ebd6f606c351e55ee22663de74371cd23855613b82b3d1a"}
Sep 18 21:00:50.098661 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:50.098306    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" event={"ID":"7b5605fc-55bf-4ef0-b90e-7da5324321af","Type":"ContainerStarted","Data":"bf8c9f47543b4ee128d1c579c75d44fc362d3e69dce445ecbcbd776dfad4272d"}
Sep 18 21:00:51.435190 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:51.435162    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 21:00:51.438975 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:51.438955    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 21:00:54.121649 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:54.121616    2583 generic.go:356] "Generic (PLEG): container finished" podID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerID="e31fcee0b80fd0982ebd6f606c351e55ee22663de74371cd23855613b82b3d1a" exitCode=0
Sep 18 21:00:54.122161 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:54.121695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" event={"ID":"7b5605fc-55bf-4ef0-b90e-7da5324321af","Type":"ContainerDied","Data":"e31fcee0b80fd0982ebd6f606c351e55ee22663de74371cd23855613b82b3d1a"}
Sep 18 21:00:55.129413 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:55.129377    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" event={"ID":"7b5605fc-55bf-4ef0-b90e-7da5324321af","Type":"ContainerStarted","Data":"c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9"}
Sep 18 21:00:55.129867 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:55.129448    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" event={"ID":"7b5605fc-55bf-4ef0-b90e-7da5324321af","Type":"ContainerStarted","Data":"7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419"}
Sep 18 21:00:55.129867 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:55.129653    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:00:55.150858 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:55.150810    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" podStartSLOduration=3.416698238 podStartE2EDuration="7.150795779s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.734097541s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:00:55.148992601 +0000 UTC m=+2406.056894888" watchObservedRunningTime="2026-09-18 21:00:55.150795779 +0000 UTC m=+2406.058698057"
Sep 18 21:00:56.133714 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:00:56.133688    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:01:02.145083 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:02.145011    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:01:32.225953 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:32.225904    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 18 21:01:42.147709 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:42.147681    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:01:48.831861 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:48.831824    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"]
Sep 18 21:01:48.832480 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:48.832057    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="kserve-container" containerID="cri-o://7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419" gracePeriod=30
Sep 18 21:01:48.832480 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:48.832162    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="kube-rbac-proxy" containerID="cri-o://c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9" gracePeriod=30
Sep 18 21:01:49.374245 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:49.374210    2583 generic.go:356] "Generic (PLEG): container finished" podID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerID="c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9" exitCode=2
Sep 18 21:01:49.374412 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:49.374287    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" event={"ID":"7b5605fc-55bf-4ef0-b90e-7da5324321af","Type":"ContainerDied","Data":"c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9"}
Sep 18 21:01:52.138573 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:52.138537    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 18 21:01:52.146395 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:52.146370    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.51:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 18 21:01:55.587740 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.587710    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:01:55.684368 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.684295    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b5605fc-55bf-4ef0-b90e-7da5324321af-cabundle-cert\") pod \"7b5605fc-55bf-4ef0-b90e-7da5324321af\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") "
Sep 18 21:01:55.684368 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.684355    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5605fc-55bf-4ef0-b90e-7da5324321af-kserve-provision-location\") pod \"7b5605fc-55bf-4ef0-b90e-7da5324321af\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") "
Sep 18 21:01:55.684575 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.684415    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b5605fc-55bf-4ef0-b90e-7da5324321af-proxy-tls\") pod \"7b5605fc-55bf-4ef0-b90e-7da5324321af\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") "
Sep 18 21:01:55.684575 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.684438    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv2bw\" (UniqueName: \"kubernetes.io/projected/7b5605fc-55bf-4ef0-b90e-7da5324321af-kube-api-access-qv2bw\") pod \"7b5605fc-55bf-4ef0-b90e-7da5324321af\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") "
Sep 18 21:01:55.684575 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.684505    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b5605fc-55bf-4ef0-b90e-7da5324321af-kube-rbac-proxy-sar-config\") pod \"7b5605fc-55bf-4ef0-b90e-7da5324321af\" (UID: \"7b5605fc-55bf-4ef0-b90e-7da5324321af\") "
Sep 18 21:01:55.684727 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.684650    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5605fc-55bf-4ef0-b90e-7da5324321af-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b5605fc-55bf-4ef0-b90e-7da5324321af" (UID: "7b5605fc-55bf-4ef0-b90e-7da5324321af"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:01:55.684727 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.684705    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5605fc-55bf-4ef0-b90e-7da5324321af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b5605fc-55bf-4ef0-b90e-7da5324321af" (UID: "7b5605fc-55bf-4ef0-b90e-7da5324321af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:01:55.684850 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.684834    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b5605fc-55bf-4ef0-b90e-7da5324321af-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:01:55.684908 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.684859    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5605fc-55bf-4ef0-b90e-7da5324321af-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:01:55.684948 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.684900    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5605fc-55bf-4ef0-b90e-7da5324321af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b5605fc-55bf-4ef0-b90e-7da5324321af" (UID: "7b5605fc-55bf-4ef0-b90e-7da5324321af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:01:55.686634 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.686615    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b5605fc-55bf-4ef0-b90e-7da5324321af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b5605fc-55bf-4ef0-b90e-7da5324321af" (UID: "7b5605fc-55bf-4ef0-b90e-7da5324321af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:01:55.686690 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.686614    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5605fc-55bf-4ef0-b90e-7da5324321af-kube-api-access-qv2bw" (OuterVolumeSpecName: "kube-api-access-qv2bw") pod "7b5605fc-55bf-4ef0-b90e-7da5324321af" (UID: "7b5605fc-55bf-4ef0-b90e-7da5324321af"). InnerVolumeSpecName "kube-api-access-qv2bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:01:55.785731 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.785696    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b5605fc-55bf-4ef0-b90e-7da5324321af-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:01:55.785731 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.785727    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv2bw\" (UniqueName: \"kubernetes.io/projected/7b5605fc-55bf-4ef0-b90e-7da5324321af-kube-api-access-qv2bw\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:01:55.785913 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:55.785739    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b5605fc-55bf-4ef0-b90e-7da5324321af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:01:56.410896 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.410864    2583 generic.go:356] "Generic (PLEG): container finished" podID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerID="7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419" exitCode=0
Sep 18 21:01:56.411057 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.410905    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" event={"ID":"7b5605fc-55bf-4ef0-b90e-7da5324321af","Type":"ContainerDied","Data":"7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419"}
Sep 18 21:01:56.411057 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.410926    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"
Sep 18 21:01:56.411057 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.410950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh" event={"ID":"7b5605fc-55bf-4ef0-b90e-7da5324321af","Type":"ContainerDied","Data":"bf8c9f47543b4ee128d1c579c75d44fc362d3e69dce445ecbcbd776dfad4272d"}
Sep 18 21:01:56.411057 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.410975    2583 scope.go:117] "RemoveContainer" containerID="c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9"
Sep 18 21:01:56.420235 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.420107    2583 scope.go:117] "RemoveContainer" containerID="7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419"
Sep 18 21:01:56.427877 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.427858    2583 scope.go:117] "RemoveContainer" containerID="e31fcee0b80fd0982ebd6f606c351e55ee22663de74371cd23855613b82b3d1a"
Sep 18 21:01:56.433268 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.433243    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"]
Sep 18 21:01:56.436447 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.436429    2583 scope.go:117] "RemoveContainer" containerID="c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9"
Sep 18 21:01:56.436762 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:01:56.436740    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9\": container with ID starting with c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9 not found: ID does not exist" containerID="c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9"
Sep 18 21:01:56.436849 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.436773    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9"} err="failed to get container status \"c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9\": rpc error: code = NotFound desc = could not find container \"c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9\": container with ID starting with c3ff57d87496ef8a2f3e083e95c240ae6ba47a2cd073a7469d3782ede89c41b9 not found: ID does not exist"
Sep 18 21:01:56.436849 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.436796    2583 scope.go:117] "RemoveContainer" containerID="7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419"
Sep 18 21:01:56.436963 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.436887    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-j92qh"]
Sep 18 21:01:56.437068 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:01:56.437053    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419\": container with ID starting with 7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419 not found: ID does not exist" containerID="7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419"
Sep 18 21:01:56.437167 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.437077    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419"} err="failed to get container status \"7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419\": rpc error: code = NotFound desc = could not find container \"7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419\": container with ID starting with 7b538bf88207a5c1d9ffd8a4e78f417b23d535f2a946be018aec382126b2f419 not found: ID does not exist"
Sep 18 21:01:56.437167 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.437100    2583 scope.go:117] "RemoveContainer" containerID="e31fcee0b80fd0982ebd6f606c351e55ee22663de74371cd23855613b82b3d1a"
Sep 18 21:01:56.437401 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:01:56.437384    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e31fcee0b80fd0982ebd6f606c351e55ee22663de74371cd23855613b82b3d1a\": container with ID starting with e31fcee0b80fd0982ebd6f606c351e55ee22663de74371cd23855613b82b3d1a not found: ID does not exist" containerID="e31fcee0b80fd0982ebd6f606c351e55ee22663de74371cd23855613b82b3d1a"
Sep 18 21:01:56.437445 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:56.437406    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e31fcee0b80fd0982ebd6f606c351e55ee22663de74371cd23855613b82b3d1a"} err="failed to get container status \"e31fcee0b80fd0982ebd6f606c351e55ee22663de74371cd23855613b82b3d1a\": rpc error: code = NotFound desc = could not find container \"e31fcee0b80fd0982ebd6f606c351e55ee22663de74371cd23855613b82b3d1a\": container with ID starting with e31fcee0b80fd0982ebd6f606c351e55ee22663de74371cd23855613b82b3d1a not found: ID does not exist"
Sep 18 21:01:57.321304 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:01:57.321270    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" path="/var/lib/kubelet/pods/7b5605fc-55bf-4ef0-b90e-7da5324321af/volumes"
Sep 18 21:02:22.058445 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.058413    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"]
Sep 18 21:02:22.058921 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.058775    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="storage-initializer"
Sep 18 21:02:22.058921 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.058790    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="storage-initializer"
Sep 18 21:02:22.058921 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.058822    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="kserve-container"
Sep 18 21:02:22.058921 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.058832    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="kserve-container"
Sep 18 21:02:22.058921 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.058845    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="kube-rbac-proxy"
Sep 18 21:02:22.058921 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.058854    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="kube-rbac-proxy"
Sep 18 21:02:22.059181 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.058967    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="kserve-container"
Sep 18 21:02:22.059181 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.058985    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5605fc-55bf-4ef0-b90e-7da5324321af" containerName="kube-rbac-proxy"
Sep 18 21:02:22.062136 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.062107    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.065622 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.065597    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 21:02:22.065748 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.065697    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 21:02:22.065813 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.065597    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 21:02:22.066274 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.066249    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 18 21:02:22.066274 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.066257    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 21:02:22.066424 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.066291    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 21:02:22.066843 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.066820    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 21:02:22.075826 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.075797    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"]
Sep 18 21:02:22.087112 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.087089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f442ff0a-58b1-40f1-b848-32ec9c601587-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.087233 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.087142    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f442ff0a-58b1-40f1-b848-32ec9c601587-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.087233 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.087210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f442ff0a-58b1-40f1-b848-32ec9c601587-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.087327 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.087245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl8mp\" (UniqueName: \"kubernetes.io/projected/f442ff0a-58b1-40f1-b848-32ec9c601587-kube-api-access-bl8mp\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.087388 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.087358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f442ff0a-58b1-40f1-b848-32ec9c601587-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.188647 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.188623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f442ff0a-58b1-40f1-b848-32ec9c601587-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.188799 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.188657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl8mp\" (UniqueName: \"kubernetes.io/projected/f442ff0a-58b1-40f1-b848-32ec9c601587-kube-api-access-bl8mp\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.188799 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.188714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f442ff0a-58b1-40f1-b848-32ec9c601587-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.188799 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.188739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f442ff0a-58b1-40f1-b848-32ec9c601587-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.188799 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.188756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f442ff0a-58b1-40f1-b848-32ec9c601587-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.188799 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:02:22.188780    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 18 21:02:22.189167 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:02:22.188841    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f442ff0a-58b1-40f1-b848-32ec9c601587-proxy-tls podName:f442ff0a-58b1-40f1-b848-32ec9c601587 nodeName:}" failed. No retries permitted until 2026-09-18 21:02:22.688824223 +0000 UTC m=+2493.596726485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f442ff0a-58b1-40f1-b848-32ec9c601587-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk" (UID: "f442ff0a-58b1-40f1-b848-32ec9c601587") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 18 21:02:22.189250 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.189213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f442ff0a-58b1-40f1-b848-32ec9c601587-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.189499 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.189478    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f442ff0a-58b1-40f1-b848-32ec9c601587-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.189566 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.189479    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f442ff0a-58b1-40f1-b848-32ec9c601587-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.199853 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.199824    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl8mp\" (UniqueName: \"kubernetes.io/projected/f442ff0a-58b1-40f1-b848-32ec9c601587-kube-api-access-bl8mp\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.693803 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.693769    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f442ff0a-58b1-40f1-b848-32ec9c601587-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.696408 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.696382    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f442ff0a-58b1-40f1-b848-32ec9c601587-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:22.981113 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:22.981027    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:23.254681 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:23.254651    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"]
Sep 18 21:02:23.257273 ip-10-0-130-79 kubenswrapper[2583]: W0918 21:02:23.257244    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf442ff0a_58b1_40f1_b848_32ec9c601587.slice/crio-d324c442eed42587b2b12397812da8ac60ac01b749a9cae61d287e2f29c9aa44 WatchSource:0}: Error finding container d324c442eed42587b2b12397812da8ac60ac01b749a9cae61d287e2f29c9aa44: Status 404 returned error can't find the container with id d324c442eed42587b2b12397812da8ac60ac01b749a9cae61d287e2f29c9aa44
Sep 18 21:02:23.536580 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:23.536492    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk" event={"ID":"f442ff0a-58b1-40f1-b848-32ec9c601587","Type":"ContainerStarted","Data":"4a68cf1914b4eda12679e61d0bd30ab1cbde7cc78a4036355e4d697316929974"}
Sep 18 21:02:23.536580 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:23.536567    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk" event={"ID":"f442ff0a-58b1-40f1-b848-32ec9c601587","Type":"ContainerStarted","Data":"d324c442eed42587b2b12397812da8ac60ac01b749a9cae61d287e2f29c9aa44"}
Sep 18 21:02:27.556637 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:27.556602    2583 generic.go:356] "Generic (PLEG): container finished" podID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerID="4a68cf1914b4eda12679e61d0bd30ab1cbde7cc78a4036355e4d697316929974" exitCode=0
Sep 18 21:02:27.557088 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:27.556642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk" event={"ID":"f442ff0a-58b1-40f1-b848-32ec9c601587","Type":"ContainerDied","Data":"4a68cf1914b4eda12679e61d0bd30ab1cbde7cc78a4036355e4d697316929974"}
Sep 18 21:02:28.563467 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:28.563430    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk" event={"ID":"f442ff0a-58b1-40f1-b848-32ec9c601587","Type":"ContainerStarted","Data":"ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4"}
Sep 18 21:02:28.563863 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:28.563514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk" event={"ID":"f442ff0a-58b1-40f1-b848-32ec9c601587","Type":"ContainerStarted","Data":"b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583"}
Sep 18 21:02:28.563863 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:28.563719    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:28.563863 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:28.563778    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:02:28.585460 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:28.585416    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk" podStartSLOduration=2.877960818 podStartE2EDuration="6.585402056s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.707441238s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:02:28.581831366 +0000 UTC m=+2499.489733647" watchObservedRunningTime="2026-09-18 21:02:28.585402056 +0000 UTC m=+2499.493304335"
Sep 18 21:02:34.573184 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:02:34.573088    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:03:04.577471 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:04.577441    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:03:12.105076 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:12.105043    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"]
Sep 18 21:03:12.105617 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:12.105322    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk" podUID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerName="kserve-container" containerID="cri-o://b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583" gracePeriod=30
Sep 18 21:03:12.105617 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:12.105357    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk" podUID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerName="kube-rbac-proxy" containerID="cri-o://ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4" gracePeriod=30
Sep 18 21:03:12.767901 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:12.767865    2583 generic.go:356] "Generic (PLEG): container finished" podID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerID="ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4" exitCode=2
Sep 18 21:03:12.768092 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:12.767939    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk" event={"ID":"f442ff0a-58b1-40f1-b848-32ec9c601587","Type":"ContainerDied","Data":"ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4"}
Sep 18 21:03:13.048185 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.048161    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:03:13.107417 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.107391    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f442ff0a-58b1-40f1-b848-32ec9c601587-kserve-provision-location\") pod \"f442ff0a-58b1-40f1-b848-32ec9c601587\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") "
Sep 18 21:03:13.107833 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.107453    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f442ff0a-58b1-40f1-b848-32ec9c601587-cabundle-cert\") pod \"f442ff0a-58b1-40f1-b848-32ec9c601587\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") "
Sep 18 21:03:13.107833 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.107479    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f442ff0a-58b1-40f1-b848-32ec9c601587-proxy-tls\") pod \"f442ff0a-58b1-40f1-b848-32ec9c601587\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") "
Sep 18 21:03:13.107833 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.107515    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl8mp\" (UniqueName: \"kubernetes.io/projected/f442ff0a-58b1-40f1-b848-32ec9c601587-kube-api-access-bl8mp\") pod \"f442ff0a-58b1-40f1-b848-32ec9c601587\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") "
Sep 18 21:03:13.107833 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.107530    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f442ff0a-58b1-40f1-b848-32ec9c601587-kube-rbac-proxy-sar-config\") pod \"f442ff0a-58b1-40f1-b848-32ec9c601587\" (UID: \"f442ff0a-58b1-40f1-b848-32ec9c601587\") "
Sep 18 21:03:13.108008 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.107839    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f442ff0a-58b1-40f1-b848-32ec9c601587-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f442ff0a-58b1-40f1-b848-32ec9c601587" (UID: "f442ff0a-58b1-40f1-b848-32ec9c601587"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:03:13.108008 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.107840    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f442ff0a-58b1-40f1-b848-32ec9c601587-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f442ff0a-58b1-40f1-b848-32ec9c601587" (UID: "f442ff0a-58b1-40f1-b848-32ec9c601587"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:03:13.108008 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.107917    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f442ff0a-58b1-40f1-b848-32ec9c601587-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f442ff0a-58b1-40f1-b848-32ec9c601587" (UID: "f442ff0a-58b1-40f1-b848-32ec9c601587"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:03:13.109684 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.109660    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f442ff0a-58b1-40f1-b848-32ec9c601587-kube-api-access-bl8mp" (OuterVolumeSpecName: "kube-api-access-bl8mp") pod "f442ff0a-58b1-40f1-b848-32ec9c601587" (UID: "f442ff0a-58b1-40f1-b848-32ec9c601587"). InnerVolumeSpecName "kube-api-access-bl8mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:03:13.109760 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.109730    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f442ff0a-58b1-40f1-b848-32ec9c601587-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f442ff0a-58b1-40f1-b848-32ec9c601587" (UID: "f442ff0a-58b1-40f1-b848-32ec9c601587"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:03:13.208993 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.208954    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f442ff0a-58b1-40f1-b848-32ec9c601587-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:03:13.208993 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.208987    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f442ff0a-58b1-40f1-b848-32ec9c601587-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:03:13.208993 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.208999    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl8mp\" (UniqueName: \"kubernetes.io/projected/f442ff0a-58b1-40f1-b848-32ec9c601587-kube-api-access-bl8mp\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:03:13.209272 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.209013    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f442ff0a-58b1-40f1-b848-32ec9c601587-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:03:13.209272 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.209025    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f442ff0a-58b1-40f1-b848-32ec9c601587-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:03:13.774533 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.774501    2583 generic.go:356] "Generic (PLEG): container finished" podID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerID="b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583" exitCode=0
Sep 18 21:03:13.774700 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.774546    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk" event={"ID":"f442ff0a-58b1-40f1-b848-32ec9c601587","Type":"ContainerDied","Data":"b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583"}
Sep 18 21:03:13.774700 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.774553    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"
Sep 18 21:03:13.774700 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.774593    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk" event={"ID":"f442ff0a-58b1-40f1-b848-32ec9c601587","Type":"ContainerDied","Data":"d324c442eed42587b2b12397812da8ac60ac01b749a9cae61d287e2f29c9aa44"}
Sep 18 21:03:13.774700 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.774621    2583 scope.go:117] "RemoveContainer" containerID="ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4"
Sep 18 21:03:13.783183 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.783163    2583 scope.go:117] "RemoveContainer" containerID="b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583"
Sep 18 21:03:13.791018 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.790992    2583 scope.go:117] "RemoveContainer" containerID="4a68cf1914b4eda12679e61d0bd30ab1cbde7cc78a4036355e4d697316929974"
Sep 18 21:03:13.792796 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.792775    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"]
Sep 18 21:03:13.796251 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.796232    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wvbmk"]
Sep 18 21:03:13.799366 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.799350    2583 scope.go:117] "RemoveContainer" containerID="ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4"
Sep 18 21:03:13.799638 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:03:13.799621    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4\": container with ID starting with ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4 not found: ID does not exist" containerID="ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4"
Sep 18 21:03:13.799714 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.799651    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4"} err="failed to get container status \"ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4\": rpc error: code = NotFound desc = could not find container \"ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4\": container with ID starting with ef30cfa1d52d39674c7c11aaf2fb93a5b23858dce20d97d6d41fd3665123dcf4 not found: ID does not exist"
Sep 18 21:03:13.799714 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.799673    2583 scope.go:117] "RemoveContainer" containerID="b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583"
Sep 18 21:03:13.799914 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:03:13.799896    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583\": container with ID starting with b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583 not found: ID does not exist" containerID="b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583"
Sep 18 21:03:13.799954 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.799920    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583"} err="failed to get container status \"b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583\": rpc error: code = NotFound desc = could not find container \"b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583\": container with ID starting with b8e82fe45add8149ea2f904e69265577983b1a4eb0aff675d64bc7a4558d1583 not found: ID does not exist"
Sep 18 21:03:13.799954 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.799935    2583 scope.go:117] "RemoveContainer" containerID="4a68cf1914b4eda12679e61d0bd30ab1cbde7cc78a4036355e4d697316929974"
Sep 18 21:03:13.800162 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:03:13.800145    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a68cf1914b4eda12679e61d0bd30ab1cbde7cc78a4036355e4d697316929974\": container with ID starting with 4a68cf1914b4eda12679e61d0bd30ab1cbde7cc78a4036355e4d697316929974 not found: ID does not exist" containerID="4a68cf1914b4eda12679e61d0bd30ab1cbde7cc78a4036355e4d697316929974"
Sep 18 21:03:13.800225 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:13.800170    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a68cf1914b4eda12679e61d0bd30ab1cbde7cc78a4036355e4d697316929974"} err="failed to get container status \"4a68cf1914b4eda12679e61d0bd30ab1cbde7cc78a4036355e4d697316929974\": rpc error: code = NotFound desc = could not find container \"4a68cf1914b4eda12679e61d0bd30ab1cbde7cc78a4036355e4d697316929974\": container with ID starting with 4a68cf1914b4eda12679e61d0bd30ab1cbde7cc78a4036355e4d697316929974 not found: ID does not exist"
Sep 18 21:03:15.315978 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:15.315944    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f442ff0a-58b1-40f1-b848-32ec9c601587" path="/var/lib/kubelet/pods/f442ff0a-58b1-40f1-b848-32ec9c601587/volumes"
Sep 18 21:03:53.393266 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.393227    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"]
Sep 18 21:03:53.393708 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.393569    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerName="storage-initializer"
Sep 18 21:03:53.393708 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.393579    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerName="storage-initializer"
Sep 18 21:03:53.393708 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.393591    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerName="kube-rbac-proxy"
Sep 18 21:03:53.393708 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.393597    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerName="kube-rbac-proxy"
Sep 18 21:03:53.393708 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.393610    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerName="kserve-container"
Sep 18 21:03:53.393708 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.393616    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerName="kserve-container"
Sep 18 21:03:53.393708 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.393667    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerName="kserve-container"
Sep 18 21:03:53.393708 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.393675    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f442ff0a-58b1-40f1-b848-32ec9c601587" containerName="kube-rbac-proxy"
Sep 18 21:03:53.397073 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.397057    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.399768 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.399746    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 21:03:53.399882 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.399752    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 18 21:03:53.399882 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.399844    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 21:03:53.399882 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.399875    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 21:03:53.400842 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.400809    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 21:03:53.400842 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.400825    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 21:03:53.401155 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.401090    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 21:03:53.408317 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.408296    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"]
Sep 18 21:03:53.549398 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.549367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb3904b-f449-4669-97da-65359187ab81-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.549398 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.549402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb3904b-f449-4669-97da-65359187ab81-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.549622 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.549424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb3904b-f449-4669-97da-65359187ab81-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.549622 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.549470    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8tg2\" (UniqueName: \"kubernetes.io/projected/3bb3904b-f449-4669-97da-65359187ab81-kube-api-access-d8tg2\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.549622 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.549533    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb3904b-f449-4669-97da-65359187ab81-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.650481 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.650393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb3904b-f449-4669-97da-65359187ab81-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.650481 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.650461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb3904b-f449-4669-97da-65359187ab81-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.650727 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.650495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb3904b-f449-4669-97da-65359187ab81-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.650727 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.650522    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb3904b-f449-4669-97da-65359187ab81-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.650727 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.650558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8tg2\" (UniqueName: \"kubernetes.io/projected/3bb3904b-f449-4669-97da-65359187ab81-kube-api-access-d8tg2\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.650899 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.650886    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb3904b-f449-4669-97da-65359187ab81-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.651270 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.651246    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb3904b-f449-4669-97da-65359187ab81-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.651362 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.651288    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb3904b-f449-4669-97da-65359187ab81-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.652952 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.652934    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb3904b-f449-4669-97da-65359187ab81-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.658997 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.658976    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8tg2\" (UniqueName: \"kubernetes.io/projected/3bb3904b-f449-4669-97da-65359187ab81-kube-api-access-d8tg2\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.708068 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.708022    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:53.984513 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:53.984481    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"]
Sep 18 21:03:53.988162 ip-10-0-130-79 kubenswrapper[2583]: W0918 21:03:53.988134    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb3904b_f449_4669_97da_65359187ab81.slice/crio-93934bd15283b654bef5fc60c56b8ae3a6420f748aeae3786e80ede1bb42f378 WatchSource:0}: Error finding container 93934bd15283b654bef5fc60c56b8ae3a6420f748aeae3786e80ede1bb42f378: Status 404 returned error can't find the container with id 93934bd15283b654bef5fc60c56b8ae3a6420f748aeae3786e80ede1bb42f378
Sep 18 21:03:54.958269 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:54.958230    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg" event={"ID":"3bb3904b-f449-4669-97da-65359187ab81","Type":"ContainerStarted","Data":"f3371383f5ccef1e41b2008ad22ad52076e501284e0ceaa3c3695520b1a78d91"}
Sep 18 21:03:54.958774 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:54.958304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg" event={"ID":"3bb3904b-f449-4669-97da-65359187ab81","Type":"ContainerStarted","Data":"93934bd15283b654bef5fc60c56b8ae3a6420f748aeae3786e80ede1bb42f378"}
Sep 18 21:03:57.973846 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:57.973758    2583 generic.go:356] "Generic (PLEG): container finished" podID="3bb3904b-f449-4669-97da-65359187ab81" containerID="f3371383f5ccef1e41b2008ad22ad52076e501284e0ceaa3c3695520b1a78d91" exitCode=0
Sep 18 21:03:57.973846 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:57.973831    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg" event={"ID":"3bb3904b-f449-4669-97da-65359187ab81","Type":"ContainerDied","Data":"f3371383f5ccef1e41b2008ad22ad52076e501284e0ceaa3c3695520b1a78d91"}
Sep 18 21:03:58.981059 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:58.981026    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg" event={"ID":"3bb3904b-f449-4669-97da-65359187ab81","Type":"ContainerStarted","Data":"450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21"}
Sep 18 21:03:58.981496 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:58.981084    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg" event={"ID":"3bb3904b-f449-4669-97da-65359187ab81","Type":"ContainerStarted","Data":"d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf"}
Sep 18 21:03:58.981496 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:58.981434    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:58.981592 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:58.981501    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:03:59.003307 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:03:59.003255    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg" podStartSLOduration=2.369400524 podStartE2EDuration="6.003236338s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.633835814s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:03:58.99977795 +0000 UTC m=+2589.907680229" watchObservedRunningTime="2026-09-18 21:03:59.003236338 +0000 UTC m=+2589.911138617"
Sep 18 21:04:04.992097 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:04.992027    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:04:34.996345 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:34.996312    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:04:43.471959 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:43.471928    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"]
Sep 18 21:04:43.472447 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:43.472275    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg" podUID="3bb3904b-f449-4669-97da-65359187ab81" containerName="kserve-container" containerID="cri-o://d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf" gracePeriod=30
Sep 18 21:04:43.472447 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:43.472304    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg" podUID="3bb3904b-f449-4669-97da-65359187ab81" containerName="kube-rbac-proxy" containerID="cri-o://450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21" gracePeriod=30
Sep 18 21:04:44.195795 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.195761    2583 generic.go:356] "Generic (PLEG): container finished" podID="3bb3904b-f449-4669-97da-65359187ab81" containerID="450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21" exitCode=2
Sep 18 21:04:44.195968 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.195802    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg" event={"ID":"3bb3904b-f449-4669-97da-65359187ab81","Type":"ContainerDied","Data":"450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21"}
Sep 18 21:04:44.617737 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.617711    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:04:44.759321 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.759227    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb3904b-f449-4669-97da-65359187ab81-kube-rbac-proxy-sar-config\") pod \"3bb3904b-f449-4669-97da-65359187ab81\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") "
Sep 18 21:04:44.759321 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.759263    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb3904b-f449-4669-97da-65359187ab81-cabundle-cert\") pod \"3bb3904b-f449-4669-97da-65359187ab81\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") "
Sep 18 21:04:44.759321 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.759309    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb3904b-f449-4669-97da-65359187ab81-kserve-provision-location\") pod \"3bb3904b-f449-4669-97da-65359187ab81\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") "
Sep 18 21:04:44.759616 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.759342    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8tg2\" (UniqueName: \"kubernetes.io/projected/3bb3904b-f449-4669-97da-65359187ab81-kube-api-access-d8tg2\") pod \"3bb3904b-f449-4669-97da-65359187ab81\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") "
Sep 18 21:04:44.759616 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.759389    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb3904b-f449-4669-97da-65359187ab81-proxy-tls\") pod \"3bb3904b-f449-4669-97da-65359187ab81\" (UID: \"3bb3904b-f449-4669-97da-65359187ab81\") "
Sep 18 21:04:44.759718 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.759695    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb3904b-f449-4669-97da-65359187ab81-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3bb3904b-f449-4669-97da-65359187ab81" (UID: "3bb3904b-f449-4669-97da-65359187ab81"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:04:44.759766 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.759734    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb3904b-f449-4669-97da-65359187ab81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3bb3904b-f449-4669-97da-65359187ab81" (UID: "3bb3904b-f449-4669-97da-65359187ab81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:04:44.759804 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.759787    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb3904b-f449-4669-97da-65359187ab81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3bb3904b-f449-4669-97da-65359187ab81" (UID: "3bb3904b-f449-4669-97da-65359187ab81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:04:44.761696 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.761667    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb3904b-f449-4669-97da-65359187ab81-kube-api-access-d8tg2" (OuterVolumeSpecName: "kube-api-access-d8tg2") pod "3bb3904b-f449-4669-97da-65359187ab81" (UID: "3bb3904b-f449-4669-97da-65359187ab81"). InnerVolumeSpecName "kube-api-access-d8tg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:04:44.761807 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.761746    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb3904b-f449-4669-97da-65359187ab81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3bb3904b-f449-4669-97da-65359187ab81" (UID: "3bb3904b-f449-4669-97da-65359187ab81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:04:44.861009 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.860978    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8tg2\" (UniqueName: \"kubernetes.io/projected/3bb3904b-f449-4669-97da-65359187ab81-kube-api-access-d8tg2\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:04:44.861009 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.861005    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb3904b-f449-4669-97da-65359187ab81-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:04:44.861009 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.861016    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb3904b-f449-4669-97da-65359187ab81-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:04:44.861250 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.861024    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb3904b-f449-4669-97da-65359187ab81-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:04:44.861250 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:44.861033    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb3904b-f449-4669-97da-65359187ab81-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:04:45.202061 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.202029    2583 generic.go:356] "Generic (PLEG): container finished" podID="3bb3904b-f449-4669-97da-65359187ab81" containerID="d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf" exitCode=0
Sep 18 21:04:45.202252 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.202084    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"
Sep 18 21:04:45.202252 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.202086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg" event={"ID":"3bb3904b-f449-4669-97da-65359187ab81","Type":"ContainerDied","Data":"d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf"}
Sep 18 21:04:45.202252 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.202194    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg" event={"ID":"3bb3904b-f449-4669-97da-65359187ab81","Type":"ContainerDied","Data":"93934bd15283b654bef5fc60c56b8ae3a6420f748aeae3786e80ede1bb42f378"}
Sep 18 21:04:45.202252 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.202233    2583 scope.go:117] "RemoveContainer" containerID="450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21"
Sep 18 21:04:45.211356 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.211337    2583 scope.go:117] "RemoveContainer" containerID="d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf"
Sep 18 21:04:45.221241 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.221060    2583 scope.go:117] "RemoveContainer" containerID="f3371383f5ccef1e41b2008ad22ad52076e501284e0ceaa3c3695520b1a78d91"
Sep 18 21:04:45.223052 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.223031    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"]
Sep 18 21:04:45.226557 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.226539    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zj4fg"]
Sep 18 21:04:45.229257 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.229240    2583 scope.go:117] "RemoveContainer" containerID="450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21"
Sep 18 21:04:45.229520 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:04:45.229501    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21\": container with ID starting with 450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21 not found: ID does not exist" containerID="450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21"
Sep 18 21:04:45.229577 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.229530    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21"} err="failed to get container status \"450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21\": rpc error: code = NotFound desc = could not find container \"450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21\": container with ID starting with 450b3fd4e8171a2f2fc333cf0dde4e98e67a9f6d821fccf64108cd277aaf9d21 not found: ID does not exist"
Sep 18 21:04:45.229577 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.229548    2583 scope.go:117] "RemoveContainer" containerID="d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf"
Sep 18 21:04:45.229782 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:04:45.229765    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf\": container with ID starting with d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf not found: ID does not exist" containerID="d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf"
Sep 18 21:04:45.229822 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.229789    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf"} err="failed to get container status \"d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf\": rpc error: code = NotFound desc = could not find container \"d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf\": container with ID starting with d76a4160154e0156ea27034eee427ad86412d299ca6c8632adf1f5fc5263d2cf not found: ID does not exist"
Sep 18 21:04:45.229822 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.229804    2583 scope.go:117] "RemoveContainer" containerID="f3371383f5ccef1e41b2008ad22ad52076e501284e0ceaa3c3695520b1a78d91"
Sep 18 21:04:45.230035 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:04:45.230019    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3371383f5ccef1e41b2008ad22ad52076e501284e0ceaa3c3695520b1a78d91\": container with ID starting with f3371383f5ccef1e41b2008ad22ad52076e501284e0ceaa3c3695520b1a78d91 not found: ID does not exist" containerID="f3371383f5ccef1e41b2008ad22ad52076e501284e0ceaa3c3695520b1a78d91"
Sep 18 21:04:45.230080 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.230040    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3371383f5ccef1e41b2008ad22ad52076e501284e0ceaa3c3695520b1a78d91"} err="failed to get container status \"f3371383f5ccef1e41b2008ad22ad52076e501284e0ceaa3c3695520b1a78d91\": rpc error: code = NotFound desc = could not find container \"f3371383f5ccef1e41b2008ad22ad52076e501284e0ceaa3c3695520b1a78d91\": container with ID starting with f3371383f5ccef1e41b2008ad22ad52076e501284e0ceaa3c3695520b1a78d91 not found: ID does not exist"
Sep 18 21:04:45.315976 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:45.315955    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bb3904b-f449-4669-97da-65359187ab81" path="/var/lib/kubelet/pods/3bb3904b-f449-4669-97da-65359187ab81/volumes"
Sep 18 21:04:48.675770 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.675726    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"]
Sep 18 21:04:48.676729 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.676500    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb3904b-f449-4669-97da-65359187ab81" containerName="kserve-container"
Sep 18 21:04:48.676729 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.676522    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb3904b-f449-4669-97da-65359187ab81" containerName="kserve-container"
Sep 18 21:04:48.676729 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.676570    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb3904b-f449-4669-97da-65359187ab81" containerName="storage-initializer"
Sep 18 21:04:48.676729 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.676580    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb3904b-f449-4669-97da-65359187ab81" containerName="storage-initializer"
Sep 18 21:04:48.676729 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.676608    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb3904b-f449-4669-97da-65359187ab81" containerName="kube-rbac-proxy"
Sep 18 21:04:48.676729 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.676618    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb3904b-f449-4669-97da-65359187ab81" containerName="kube-rbac-proxy"
Sep 18 21:04:48.677163 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.676747    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb3904b-f449-4669-97da-65359187ab81" containerName="kube-rbac-proxy"
Sep 18 21:04:48.677163 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.676761    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb3904b-f449-4669-97da-65359187ab81" containerName="kserve-container"
Sep 18 21:04:48.680066 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.680044    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.682667 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.682646    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 18 21:04:48.682849 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.682836    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 21:04:48.682912 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.682855    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 21:04:48.683003 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.682985    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 21:04:48.683003 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.682995    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 21:04:48.683209 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.683033    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 18 21:04:48.683802 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.683787    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 21:04:48.689853 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.689833    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"]
Sep 18 21:04:48.795881 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.795836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2f10bcc-be69-405e-9e56-98ee3351937a-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.795881 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.795875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2f10bcc-be69-405e-9e56-98ee3351937a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.796178 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.795969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2f10bcc-be69-405e-9e56-98ee3351937a-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.796178 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.796034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcbl6\" (UniqueName: \"kubernetes.io/projected/b2f10bcc-be69-405e-9e56-98ee3351937a-kube-api-access-zcbl6\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.796178 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.796088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2f10bcc-be69-405e-9e56-98ee3351937a-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.897308 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.897275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2f10bcc-be69-405e-9e56-98ee3351937a-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.897454 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.897320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcbl6\" (UniqueName: \"kubernetes.io/projected/b2f10bcc-be69-405e-9e56-98ee3351937a-kube-api-access-zcbl6\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.897454 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.897352    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2f10bcc-be69-405e-9e56-98ee3351937a-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.897454 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.897381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2f10bcc-be69-405e-9e56-98ee3351937a-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.897454 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.897398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2f10bcc-be69-405e-9e56-98ee3351937a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.897854 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.897832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2f10bcc-be69-405e-9e56-98ee3351937a-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.898007 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.897986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2f10bcc-be69-405e-9e56-98ee3351937a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.898068 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.898052    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2f10bcc-be69-405e-9e56-98ee3351937a-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.900063 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.900046    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2f10bcc-be69-405e-9e56-98ee3351937a-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.905773 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.905744    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcbl6\" (UniqueName: \"kubernetes.io/projected/b2f10bcc-be69-405e-9e56-98ee3351937a-kube-api-access-zcbl6\") pod \"isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:48.990838 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:48.990767    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:49.266671 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:49.266647    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"]
Sep 18 21:04:49.268287 ip-10-0-130-79 kubenswrapper[2583]: W0918 21:04:49.268260    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f10bcc_be69_405e_9e56_98ee3351937a.slice/crio-12fb714f787ae06fbe0ae65aaf589e5fd0a89d93f993ddf0bc0b68ce460989b5 WatchSource:0}: Error finding container 12fb714f787ae06fbe0ae65aaf589e5fd0a89d93f993ddf0bc0b68ce460989b5: Status 404 returned error can't find the container with id 12fb714f787ae06fbe0ae65aaf589e5fd0a89d93f993ddf0bc0b68ce460989b5
Sep 18 21:04:50.228574 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:50.228539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" event={"ID":"b2f10bcc-be69-405e-9e56-98ee3351937a","Type":"ContainerStarted","Data":"a746350ea296b17dba6f56cd4fe529a79a8b2c17bddf7b256d6e7b4957245a1a"}
Sep 18 21:04:50.228993 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:50.228602    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" event={"ID":"b2f10bcc-be69-405e-9e56-98ee3351937a","Type":"ContainerStarted","Data":"12fb714f787ae06fbe0ae65aaf589e5fd0a89d93f993ddf0bc0b68ce460989b5"}
Sep 18 21:04:54.248840 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:54.248804    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerID="a746350ea296b17dba6f56cd4fe529a79a8b2c17bddf7b256d6e7b4957245a1a" exitCode=0
Sep 18 21:04:54.249269 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:54.248875    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" event={"ID":"b2f10bcc-be69-405e-9e56-98ee3351937a","Type":"ContainerDied","Data":"a746350ea296b17dba6f56cd4fe529a79a8b2c17bddf7b256d6e7b4957245a1a"}
Sep 18 21:04:54.320667 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:54.320648    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:04:55.258641 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:55.258602    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" event={"ID":"b2f10bcc-be69-405e-9e56-98ee3351937a","Type":"ContainerStarted","Data":"2996aacd18d33aae059e5d8cd21117c8086d7f657c87b7ab49ed206428a5d714"}
Sep 18 21:04:57.270838 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:57.270809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" event={"ID":"b2f10bcc-be69-405e-9e56-98ee3351937a","Type":"ContainerStarted","Data":"ba02bd943b9ef721c277fde4e6420c8b00aa4f54e72b880fd12b685ff0be4f38"}
Sep 18 21:04:58.277515 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:58.277479    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" event={"ID":"b2f10bcc-be69-405e-9e56-98ee3351937a","Type":"ContainerStarted","Data":"b4dd4d175cdbe999d7c0c625a1c9e8ba455c3aa8b1d90c0573b4843d9728fff0"}
Sep 18 21:04:58.277878 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:58.277617    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:58.277878 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:58.277663    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:04:58.300544 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:58.300497    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podStartSLOduration=3.537420749 podStartE2EDuration="10.300481653s" totalImagesPullingTime="2.826720379s" totalInitContainerRuntime="3.936340525s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:04:58.297358172 +0000 UTC m=+2649.205260448" watchObservedRunningTime="2026-09-18 21:04:58.300481653 +0000 UTC m=+2649.208383931"
Sep 18 21:04:59.283355 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:04:59.283325    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:05:05.292955 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:05:05.292925    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:05:25.294741 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:05:25.294704    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 21:05:28.991390 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:05:28.991340    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 21:05:35.434083 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:05:35.433998    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:05:51.465647 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:05:51.465621    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 21:05:51.469989 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:05:51.469971    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 21:06:05.435778 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:05.435745    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:06:08.711257 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:08.711222    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"]
Sep 18 21:06:08.711731 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:08.711484    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-container" containerID="cri-o://2996aacd18d33aae059e5d8cd21117c8086d7f657c87b7ab49ed206428a5d714" gracePeriod=30
Sep 18 21:06:08.711731 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:08.711535    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-agent" containerID="cri-o://ba02bd943b9ef721c277fde4e6420c8b00aa4f54e72b880fd12b685ff0be4f38" gracePeriod=30
Sep 18 21:06:08.711731 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:08.711535    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kube-rbac-proxy" containerID="cri-o://b4dd4d175cdbe999d7c0c625a1c9e8ba455c3aa8b1d90c0573b4843d9728fff0" gracePeriod=30
Sep 18 21:06:09.608373 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:09.608340    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerID="b4dd4d175cdbe999d7c0c625a1c9e8ba455c3aa8b1d90c0573b4843d9728fff0" exitCode=2
Sep 18 21:06:09.608532 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:09.608408    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" event={"ID":"b2f10bcc-be69-405e-9e56-98ee3351937a","Type":"ContainerDied","Data":"b4dd4d175cdbe999d7c0c625a1c9e8ba455c3aa8b1d90c0573b4843d9728fff0"}
Sep 18 21:06:10.288741 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:10.288701    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 18 21:06:11.620297 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:11.620266    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerID="2996aacd18d33aae059e5d8cd21117c8086d7f657c87b7ab49ed206428a5d714" exitCode=0
Sep 18 21:06:11.620700 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:11.620342    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" event={"ID":"b2f10bcc-be69-405e-9e56-98ee3351937a","Type":"ContainerDied","Data":"2996aacd18d33aae059e5d8cd21117c8086d7f657c87b7ab49ed206428a5d714"}
Sep 18 21:06:15.287927 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:15.287880    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 18 21:06:15.294156 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:15.294105    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 21:06:20.288595 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:20.288546    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 18 21:06:20.289035 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:20.288691    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:06:25.288080 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:25.288033    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 18 21:06:25.294072 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:25.294047    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 21:06:30.288372 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:30.288326    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 18 21:06:35.288369 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:35.288329    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 18 21:06:35.293801 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:35.293777    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 21:06:35.293888 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:35.293876    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:06:38.757396 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:38.757363    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerID="ba02bd943b9ef721c277fde4e6420c8b00aa4f54e72b880fd12b685ff0be4f38" exitCode=0
Sep 18 21:06:38.757830 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:38.757429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" event={"ID":"b2f10bcc-be69-405e-9e56-98ee3351937a","Type":"ContainerDied","Data":"ba02bd943b9ef721c277fde4e6420c8b00aa4f54e72b880fd12b685ff0be4f38"}
Sep 18 21:06:38.869624 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:38.869600    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:06:39.013511 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.013429    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2f10bcc-be69-405e-9e56-98ee3351937a-proxy-tls\") pod \"b2f10bcc-be69-405e-9e56-98ee3351937a\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") "
Sep 18 21:06:39.013511 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.013471    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2f10bcc-be69-405e-9e56-98ee3351937a-kube-rbac-proxy-sar-config\") pod \"b2f10bcc-be69-405e-9e56-98ee3351937a\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") "
Sep 18 21:06:39.013511 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.013491    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2f10bcc-be69-405e-9e56-98ee3351937a-kserve-provision-location\") pod \"b2f10bcc-be69-405e-9e56-98ee3351937a\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") "
Sep 18 21:06:39.013775 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.013527    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcbl6\" (UniqueName: \"kubernetes.io/projected/b2f10bcc-be69-405e-9e56-98ee3351937a-kube-api-access-zcbl6\") pod \"b2f10bcc-be69-405e-9e56-98ee3351937a\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") "
Sep 18 21:06:39.013775 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.013570    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2f10bcc-be69-405e-9e56-98ee3351937a-cabundle-cert\") pod \"b2f10bcc-be69-405e-9e56-98ee3351937a\" (UID: \"b2f10bcc-be69-405e-9e56-98ee3351937a\") "
Sep 18 21:06:39.013892 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.013823    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f10bcc-be69-405e-9e56-98ee3351937a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2f10bcc-be69-405e-9e56-98ee3351937a" (UID: "b2f10bcc-be69-405e-9e56-98ee3351937a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:06:39.013955 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.013830    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f10bcc-be69-405e-9e56-98ee3351937a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2f10bcc-be69-405e-9e56-98ee3351937a" (UID: "b2f10bcc-be69-405e-9e56-98ee3351937a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:06:39.014041 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.014012    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f10bcc-be69-405e-9e56-98ee3351937a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b2f10bcc-be69-405e-9e56-98ee3351937a" (UID: "b2f10bcc-be69-405e-9e56-98ee3351937a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:06:39.015773 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.015745    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f10bcc-be69-405e-9e56-98ee3351937a-kube-api-access-zcbl6" (OuterVolumeSpecName: "kube-api-access-zcbl6") pod "b2f10bcc-be69-405e-9e56-98ee3351937a" (UID: "b2f10bcc-be69-405e-9e56-98ee3351937a"). InnerVolumeSpecName "kube-api-access-zcbl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:06:39.015773 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.015770    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f10bcc-be69-405e-9e56-98ee3351937a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2f10bcc-be69-405e-9e56-98ee3351937a" (UID: "b2f10bcc-be69-405e-9e56-98ee3351937a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:06:39.114482 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.114451    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2f10bcc-be69-405e-9e56-98ee3351937a-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:06:39.114482 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.114479    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2f10bcc-be69-405e-9e56-98ee3351937a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:06:39.114482 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.114488    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2f10bcc-be69-405e-9e56-98ee3351937a-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:06:39.114687 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.114498    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcbl6\" (UniqueName: \"kubernetes.io/projected/b2f10bcc-be69-405e-9e56-98ee3351937a-kube-api-access-zcbl6\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:06:39.114687 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.114506    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2f10bcc-be69-405e-9e56-98ee3351937a-cabundle-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:06:39.764750 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.764698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt" event={"ID":"b2f10bcc-be69-405e-9e56-98ee3351937a","Type":"ContainerDied","Data":"12fb714f787ae06fbe0ae65aaf589e5fd0a89d93f993ddf0bc0b68ce460989b5"}
Sep 18 21:06:39.765219 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.764789    2583 scope.go:117] "RemoveContainer" containerID="b4dd4d175cdbe999d7c0c625a1c9e8ba455c3aa8b1d90c0573b4843d9728fff0"
Sep 18 21:06:39.765219 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.764731    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"
Sep 18 21:06:39.773405 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.773384    2583 scope.go:117] "RemoveContainer" containerID="ba02bd943b9ef721c277fde4e6420c8b00aa4f54e72b880fd12b685ff0be4f38"
Sep 18 21:06:39.780839 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.780823    2583 scope.go:117] "RemoveContainer" containerID="2996aacd18d33aae059e5d8cd21117c8086d7f657c87b7ab49ed206428a5d714"
Sep 18 21:06:39.784262 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.784238    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"]
Sep 18 21:06:39.787311 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.787294    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7ffd8fb554-j85rt"]
Sep 18 21:06:39.791283 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:39.791266    2583 scope.go:117] "RemoveContainer" containerID="a746350ea296b17dba6f56cd4fe529a79a8b2c17bddf7b256d6e7b4957245a1a"
Sep 18 21:06:41.315933 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:06:41.315902    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" path="/var/lib/kubelet/pods/b2f10bcc-be69-405e-9e56-98ee3351937a/volumes"
Sep 18 21:07:54.052284 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.052250    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"]
Sep 18 21:07:54.052832 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.052627    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-agent"
Sep 18 21:07:54.052832 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.052641    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-agent"
Sep 18 21:07:54.052832 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.052654    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kube-rbac-proxy"
Sep 18 21:07:54.052832 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.052662    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kube-rbac-proxy"
Sep 18 21:07:54.052832 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.052684    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-container"
Sep 18 21:07:54.052832 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.052690    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-container"
Sep 18 21:07:54.052832 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.052700    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="storage-initializer"
Sep 18 21:07:54.052832 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.052709    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="storage-initializer"
Sep 18 21:07:54.052832 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.052785    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-agent"
Sep 18 21:07:54.052832 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.052797    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kserve-container"
Sep 18 21:07:54.052832 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.052809    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f10bcc-be69-405e-9e56-98ee3351937a" containerName="kube-rbac-proxy"
Sep 18 21:07:54.055658 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.055643    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.058641 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.058612    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 18 21:07:54.058747 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.058637    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 21:07:54.058747 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.058703    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 21:07:54.058747 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.058637    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 21:07:54.058920 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.058637    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 21:07:54.059544 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.059524    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 18 21:07:54.070328 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.070304    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"]
Sep 18 21:07:54.140170 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.140135    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0559700-4ff8-4935-99f7-c2a804314577-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.140326 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.140202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0559700-4ff8-4935-99f7-c2a804314577-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.140326 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.140222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjmfw\" (UniqueName: \"kubernetes.io/projected/f0559700-4ff8-4935-99f7-c2a804314577-kube-api-access-jjmfw\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.140326 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.140250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0559700-4ff8-4935-99f7-c2a804314577-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.241409 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.241361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0559700-4ff8-4935-99f7-c2a804314577-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.241601 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.241459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0559700-4ff8-4935-99f7-c2a804314577-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.241601 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.241490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjmfw\" (UniqueName: \"kubernetes.io/projected/f0559700-4ff8-4935-99f7-c2a804314577-kube-api-access-jjmfw\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.241723 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.241638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0559700-4ff8-4935-99f7-c2a804314577-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.242013 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.241984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0559700-4ff8-4935-99f7-c2a804314577-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.242190 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.242171    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0559700-4ff8-4935-99f7-c2a804314577-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.243974 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.243959    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0559700-4ff8-4935-99f7-c2a804314577-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.249614 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.249591    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjmfw\" (UniqueName: \"kubernetes.io/projected/f0559700-4ff8-4935-99f7-c2a804314577-kube-api-access-jjmfw\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.366607 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.366523    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:07:54.640910 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:54.640882    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"]
Sep 18 21:07:54.642630 ip-10-0-130-79 kubenswrapper[2583]: W0918 21:07:54.642603    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0559700_4ff8_4935_99f7_c2a804314577.slice/crio-004c165e6b79545f41fe8837133898a2c7acc1a7dce53b18c695f98e61ce2f62 WatchSource:0}: Error finding container 004c165e6b79545f41fe8837133898a2c7acc1a7dce53b18c695f98e61ce2f62: Status 404 returned error can't find the container with id 004c165e6b79545f41fe8837133898a2c7acc1a7dce53b18c695f98e61ce2f62
Sep 18 21:07:55.111945 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:55.111911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7" event={"ID":"f0559700-4ff8-4935-99f7-c2a804314577","Type":"ContainerStarted","Data":"78100b4b0a0d70a4079346bdf913faace201fc664058f440f45b8d70de46495f"}
Sep 18 21:07:55.112344 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:55.111996    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7" event={"ID":"f0559700-4ff8-4935-99f7-c2a804314577","Type":"ContainerStarted","Data":"004c165e6b79545f41fe8837133898a2c7acc1a7dce53b18c695f98e61ce2f62"}
Sep 18 21:07:59.131074 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:59.131047    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7_f0559700-4ff8-4935-99f7-c2a804314577/storage-initializer/0.log"
Sep 18 21:07:59.131568 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:59.131084    2583 generic.go:356] "Generic (PLEG): container finished" podID="f0559700-4ff8-4935-99f7-c2a804314577" containerID="78100b4b0a0d70a4079346bdf913faace201fc664058f440f45b8d70de46495f" exitCode=1
Sep 18 21:07:59.131568 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:07:59.131173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7" event={"ID":"f0559700-4ff8-4935-99f7-c2a804314577","Type":"ContainerDied","Data":"78100b4b0a0d70a4079346bdf913faace201fc664058f440f45b8d70de46495f"}
Sep 18 21:08:00.137508 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:00.137483    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7_f0559700-4ff8-4935-99f7-c2a804314577/storage-initializer/0.log"
Sep 18 21:08:00.137981 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:00.137558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7" event={"ID":"f0559700-4ff8-4935-99f7-c2a804314577","Type":"ContainerStarted","Data":"9574718c1f27b60b784d6e7fece2b5b1c79c08b4b5ed45af589f2b1f7a65db91"}
Sep 18 21:08:04.026078 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.026047    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"]
Sep 18 21:08:04.026521 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.026272    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7" podUID="f0559700-4ff8-4935-99f7-c2a804314577" containerName="storage-initializer" containerID="cri-o://9574718c1f27b60b784d6e7fece2b5b1c79c08b4b5ed45af589f2b1f7a65db91" gracePeriod=30
Sep 18 21:08:04.157880 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.157852    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7_f0559700-4ff8-4935-99f7-c2a804314577/storage-initializer/1.log"
Sep 18 21:08:04.158307 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.158288    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7_f0559700-4ff8-4935-99f7-c2a804314577/storage-initializer/0.log"
Sep 18 21:08:04.158425 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.158324    2583 generic.go:356] "Generic (PLEG): container finished" podID="f0559700-4ff8-4935-99f7-c2a804314577" containerID="9574718c1f27b60b784d6e7fece2b5b1c79c08b4b5ed45af589f2b1f7a65db91" exitCode=1
Sep 18 21:08:04.158425 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.158400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7" event={"ID":"f0559700-4ff8-4935-99f7-c2a804314577","Type":"ContainerDied","Data":"9574718c1f27b60b784d6e7fece2b5b1c79c08b4b5ed45af589f2b1f7a65db91"}
Sep 18 21:08:04.158536 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.158461    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7" event={"ID":"f0559700-4ff8-4935-99f7-c2a804314577","Type":"ContainerDied","Data":"004c165e6b79545f41fe8837133898a2c7acc1a7dce53b18c695f98e61ce2f62"}
Sep 18 21:08:04.158536 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.158482    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="004c165e6b79545f41fe8837133898a2c7acc1a7dce53b18c695f98e61ce2f62"
Sep 18 21:08:04.158536 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.158495    2583 scope.go:117] "RemoveContainer" containerID="78100b4b0a0d70a4079346bdf913faace201fc664058f440f45b8d70de46495f"
Sep 18 21:08:04.161802 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.161786    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7_f0559700-4ff8-4935-99f7-c2a804314577/storage-initializer/1.log"
Sep 18 21:08:04.164519 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.164502    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:08:04.224010 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.223984    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0559700-4ff8-4935-99f7-c2a804314577-kserve-provision-location\") pod \"f0559700-4ff8-4935-99f7-c2a804314577\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") "
Sep 18 21:08:04.224117 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.224029    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjmfw\" (UniqueName: \"kubernetes.io/projected/f0559700-4ff8-4935-99f7-c2a804314577-kube-api-access-jjmfw\") pod \"f0559700-4ff8-4935-99f7-c2a804314577\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") "
Sep 18 21:08:04.224117 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.224066    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0559700-4ff8-4935-99f7-c2a804314577-proxy-tls\") pod \"f0559700-4ff8-4935-99f7-c2a804314577\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") "
Sep 18 21:08:04.224269 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.224162    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0559700-4ff8-4935-99f7-c2a804314577-kube-rbac-proxy-sar-config\") pod \"f0559700-4ff8-4935-99f7-c2a804314577\" (UID: \"f0559700-4ff8-4935-99f7-c2a804314577\") "
Sep 18 21:08:04.224329 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.224262    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0559700-4ff8-4935-99f7-c2a804314577-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0559700-4ff8-4935-99f7-c2a804314577" (UID: "f0559700-4ff8-4935-99f7-c2a804314577"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:08:04.224424 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.224408    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0559700-4ff8-4935-99f7-c2a804314577-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:08:04.224584 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.224566    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0559700-4ff8-4935-99f7-c2a804314577-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0559700-4ff8-4935-99f7-c2a804314577" (UID: "f0559700-4ff8-4935-99f7-c2a804314577"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:08:04.226159 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.226134    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0559700-4ff8-4935-99f7-c2a804314577-kube-api-access-jjmfw" (OuterVolumeSpecName: "kube-api-access-jjmfw") pod "f0559700-4ff8-4935-99f7-c2a804314577" (UID: "f0559700-4ff8-4935-99f7-c2a804314577"). InnerVolumeSpecName "kube-api-access-jjmfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:08:04.226250 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.226214    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0559700-4ff8-4935-99f7-c2a804314577-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0559700-4ff8-4935-99f7-c2a804314577" (UID: "f0559700-4ff8-4935-99f7-c2a804314577"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:08:04.325149 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.325078    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0559700-4ff8-4935-99f7-c2a804314577-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:08:04.325149 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.325098    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjmfw\" (UniqueName: \"kubernetes.io/projected/f0559700-4ff8-4935-99f7-c2a804314577-kube-api-access-jjmfw\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:08:04.325149 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:04.325108    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0559700-4ff8-4935-99f7-c2a804314577-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:08:05.164635 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:05.164608    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7_f0559700-4ff8-4935-99f7-c2a804314577/storage-initializer/1.log"
Sep 18 21:08:05.165073 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:05.164705    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"
Sep 18 21:08:05.210636 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:05.210597    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"]
Sep 18 21:08:05.212790 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:05.212766    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-bpwn7"]
Sep 18 21:08:05.315886 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:08:05.315858    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0559700-4ff8-4935-99f7-c2a804314577" path="/var/lib/kubelet/pods/f0559700-4ff8-4935-99f7-c2a804314577/volumes"
Sep 18 21:09:36.288478 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.288444    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"]
Sep 18 21:09:36.288953 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.288793    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0559700-4ff8-4935-99f7-c2a804314577" containerName="storage-initializer"
Sep 18 21:09:36.288953 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.288804    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0559700-4ff8-4935-99f7-c2a804314577" containerName="storage-initializer"
Sep 18 21:09:36.288953 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.288818    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0559700-4ff8-4935-99f7-c2a804314577" containerName="storage-initializer"
Sep 18 21:09:36.288953 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.288823    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0559700-4ff8-4935-99f7-c2a804314577" containerName="storage-initializer"
Sep 18 21:09:36.288953 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.288886    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0559700-4ff8-4935-99f7-c2a804314577" containerName="storage-initializer"
Sep 18 21:09:36.288953 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.288895    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0559700-4ff8-4935-99f7-c2a804314577" containerName="storage-initializer"
Sep 18 21:09:36.291924 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.291904    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.294706 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.294682    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 21:09:36.294854 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.294685    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 18 21:09:36.295803 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.295780    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 18 21:09:36.295925 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.295812    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 21:09:36.295925 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.295813    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 21:09:36.296116 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.296098    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 21:09:36.302952 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.302930    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"]
Sep 18 21:09:36.492992 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.492947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7e78759-1efd-4106-b995-d945e25daa2b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.492992 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.493000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e78759-1efd-4106-b995-d945e25daa2b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.493235 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.493045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e78759-1efd-4106-b995-d945e25daa2b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.493235 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.493106    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnlnm\" (UniqueName: \"kubernetes.io/projected/f7e78759-1efd-4106-b995-d945e25daa2b-kube-api-access-wnlnm\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.593770 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.593688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7e78759-1efd-4106-b995-d945e25daa2b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.593770 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.593724    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e78759-1efd-4106-b995-d945e25daa2b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.593770 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.593752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e78759-1efd-4106-b995-d945e25daa2b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.594040 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.593811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnlnm\" (UniqueName: \"kubernetes.io/projected/f7e78759-1efd-4106-b995-d945e25daa2b-kube-api-access-wnlnm\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.594191 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.594165    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7e78759-1efd-4106-b995-d945e25daa2b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.594410 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.594392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e78759-1efd-4106-b995-d945e25daa2b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.596279 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.596260    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e78759-1efd-4106-b995-d945e25daa2b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.601864 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.601828    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnlnm\" (UniqueName: \"kubernetes.io/projected/f7e78759-1efd-4106-b995-d945e25daa2b-kube-api-access-wnlnm\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.601997 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.601978    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:36.871541 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:36.871512    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"]
Sep 18 21:09:36.873704 ip-10-0-130-79 kubenswrapper[2583]: W0918 21:09:36.873641    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e78759_1efd_4106_b995_d945e25daa2b.slice/crio-47d29ef4247e5601f30917c1b7063a2b43474e34085a9d29ed2ea0052b09e860 WatchSource:0}: Error finding container 47d29ef4247e5601f30917c1b7063a2b43474e34085a9d29ed2ea0052b09e860: Status 404 returned error can't find the container with id 47d29ef4247e5601f30917c1b7063a2b43474e34085a9d29ed2ea0052b09e860
Sep 18 21:09:37.582497 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:37.582458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g" event={"ID":"f7e78759-1efd-4106-b995-d945e25daa2b","Type":"ContainerStarted","Data":"3fb1ef3b96950caf231e9b583d3c907e05f43333b5aecb9bd77544e123ca3ca4"}
Sep 18 21:09:37.582880 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:37.582528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g" event={"ID":"f7e78759-1efd-4106-b995-d945e25daa2b","Type":"ContainerStarted","Data":"47d29ef4247e5601f30917c1b7063a2b43474e34085a9d29ed2ea0052b09e860"}
Sep 18 21:09:39.599382 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:39.599304    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g_f7e78759-1efd-4106-b995-d945e25daa2b/storage-initializer/0.log"
Sep 18 21:09:39.599382 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:39.599343    2583 generic.go:356] "Generic (PLEG): container finished" podID="f7e78759-1efd-4106-b995-d945e25daa2b" containerID="3fb1ef3b96950caf231e9b583d3c907e05f43333b5aecb9bd77544e123ca3ca4" exitCode=1
Sep 18 21:09:39.599382 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:39.599372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g" event={"ID":"f7e78759-1efd-4106-b995-d945e25daa2b","Type":"ContainerDied","Data":"3fb1ef3b96950caf231e9b583d3c907e05f43333b5aecb9bd77544e123ca3ca4"}
Sep 18 21:09:40.607678 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:40.607650    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g_f7e78759-1efd-4106-b995-d945e25daa2b/storage-initializer/0.log"
Sep 18 21:09:40.608152 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:40.607732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g" event={"ID":"f7e78759-1efd-4106-b995-d945e25daa2b","Type":"ContainerStarted","Data":"656e3b632c7465857040dde6f4630963e3b1641ddd5c365a56548cb75e370182"}
Sep 18 21:09:42.619601 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:42.619526    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g_f7e78759-1efd-4106-b995-d945e25daa2b/storage-initializer/1.log"
Sep 18 21:09:42.619965 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:42.619903    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g_f7e78759-1efd-4106-b995-d945e25daa2b/storage-initializer/0.log"
Sep 18 21:09:42.619965 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:42.619936    2583 generic.go:356] "Generic (PLEG): container finished" podID="f7e78759-1efd-4106-b995-d945e25daa2b" containerID="656e3b632c7465857040dde6f4630963e3b1641ddd5c365a56548cb75e370182" exitCode=1
Sep 18 21:09:42.620054 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:42.620016    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g" event={"ID":"f7e78759-1efd-4106-b995-d945e25daa2b","Type":"ContainerDied","Data":"656e3b632c7465857040dde6f4630963e3b1641ddd5c365a56548cb75e370182"}
Sep 18 21:09:42.620101 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:42.620079    2583 scope.go:117] "RemoveContainer" containerID="3fb1ef3b96950caf231e9b583d3c907e05f43333b5aecb9bd77544e123ca3ca4"
Sep 18 21:09:42.620539 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:42.620506    2583 scope.go:117] "RemoveContainer" containerID="3fb1ef3b96950caf231e9b583d3c907e05f43333b5aecb9bd77544e123ca3ca4"
Sep 18 21:09:42.630963 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:09:42.630940    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g_kserve-ci-e2e-test(f7e78759-1efd-4106-b995-d945e25daa2b)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g" podUID="f7e78759-1efd-4106-b995-d945e25daa2b"
Sep 18 21:09:42.631057 ip-10-0-130-79 kubenswrapper[2583]: E0918 21:09:42.630966    2583 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g_kserve-ci-e2e-test_f7e78759-1efd-4106-b995-d945e25daa2b_0 in pod sandbox 47d29ef4247e5601f30917c1b7063a2b43474e34085a9d29ed2ea0052b09e860 from index: no such id: '3fb1ef3b96950caf231e9b583d3c907e05f43333b5aecb9bd77544e123ca3ca4'" containerID="3fb1ef3b96950caf231e9b583d3c907e05f43333b5aecb9bd77544e123ca3ca4"
Sep 18 21:09:42.631057 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:42.631004    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb1ef3b96950caf231e9b583d3c907e05f43333b5aecb9bd77544e123ca3ca4"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g_kserve-ci-e2e-test_f7e78759-1efd-4106-b995-d945e25daa2b_0 in pod sandbox 47d29ef4247e5601f30917c1b7063a2b43474e34085a9d29ed2ea0052b09e860 from index: no such id: '3fb1ef3b96950caf231e9b583d3c907e05f43333b5aecb9bd77544e123ca3ca4'"
Sep 18 21:09:43.626769 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:43.626744    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g_f7e78759-1efd-4106-b995-d945e25daa2b/storage-initializer/1.log"
Sep 18 21:09:46.252669 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.252638    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"]
Sep 18 21:09:46.382891 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.382871    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g_f7e78759-1efd-4106-b995-d945e25daa2b/storage-initializer/1.log"
Sep 18 21:09:46.386293 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.386277    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:46.467700 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.467676    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e78759-1efd-4106-b995-d945e25daa2b-proxy-tls\") pod \"f7e78759-1efd-4106-b995-d945e25daa2b\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") "
Sep 18 21:09:46.467819 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.467706    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7e78759-1efd-4106-b995-d945e25daa2b-kserve-provision-location\") pod \"f7e78759-1efd-4106-b995-d945e25daa2b\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") "
Sep 18 21:09:46.467819 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.467746    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnlnm\" (UniqueName: \"kubernetes.io/projected/f7e78759-1efd-4106-b995-d945e25daa2b-kube-api-access-wnlnm\") pod \"f7e78759-1efd-4106-b995-d945e25daa2b\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") "
Sep 18 21:09:46.467819 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.467770    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e78759-1efd-4106-b995-d945e25daa2b-kube-rbac-proxy-sar-config\") pod \"f7e78759-1efd-4106-b995-d945e25daa2b\" (UID: \"f7e78759-1efd-4106-b995-d945e25daa2b\") "
Sep 18 21:09:46.468021 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.467968    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e78759-1efd-4106-b995-d945e25daa2b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7e78759-1efd-4106-b995-d945e25daa2b" (UID: "f7e78759-1efd-4106-b995-d945e25daa2b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:09:46.468160 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.468110    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e78759-1efd-4106-b995-d945e25daa2b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f7e78759-1efd-4106-b995-d945e25daa2b" (UID: "f7e78759-1efd-4106-b995-d945e25daa2b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:09:46.469856 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.469832    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e78759-1efd-4106-b995-d945e25daa2b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7e78759-1efd-4106-b995-d945e25daa2b" (UID: "f7e78759-1efd-4106-b995-d945e25daa2b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:09:46.469932 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.469909    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e78759-1efd-4106-b995-d945e25daa2b-kube-api-access-wnlnm" (OuterVolumeSpecName: "kube-api-access-wnlnm") pod "f7e78759-1efd-4106-b995-d945e25daa2b" (UID: "f7e78759-1efd-4106-b995-d945e25daa2b"). InnerVolumeSpecName "kube-api-access-wnlnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:09:46.568538 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.568451    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e78759-1efd-4106-b995-d945e25daa2b-proxy-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:09:46.568538 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.568492    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7e78759-1efd-4106-b995-d945e25daa2b-kserve-provision-location\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:09:46.568538 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.568502    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnlnm\" (UniqueName: \"kubernetes.io/projected/f7e78759-1efd-4106-b995-d945e25daa2b-kube-api-access-wnlnm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:09:46.568538 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.568510    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e78759-1efd-4106-b995-d945e25daa2b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\""
Sep 18 21:09:46.641987 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.641963    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g_f7e78759-1efd-4106-b995-d945e25daa2b/storage-initializer/1.log"
Sep 18 21:09:46.642146 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.642051    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"
Sep 18 21:09:46.642146 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.642064    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g" event={"ID":"f7e78759-1efd-4106-b995-d945e25daa2b","Type":"ContainerDied","Data":"47d29ef4247e5601f30917c1b7063a2b43474e34085a9d29ed2ea0052b09e860"}
Sep 18 21:09:46.642230 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.642165    2583 scope.go:117] "RemoveContainer" containerID="656e3b632c7465857040dde6f4630963e3b1641ddd5c365a56548cb75e370182"
Sep 18 21:09:46.675227 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.675197    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"]
Sep 18 21:09:46.678760 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:46.678734    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6646ddb8bb-2sg4g"]
Sep 18 21:09:47.316491 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:09:47.316462    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e78759-1efd-4106-b995-d945e25daa2b" path="/var/lib/kubelet/pods/f7e78759-1efd-4106-b995-d945e25daa2b/volumes"
Sep 18 21:10:51.500686 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:10:51.500653    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 21:10:51.505762 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:10:51.505742    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 21:14:15.501250 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:15.501221    2583 scope.go:117] "RemoveContainer" containerID="9574718c1f27b60b784d6e7fece2b5b1c79c08b4b5ed45af589f2b1f7a65db91"
Sep 18 21:14:20.319156 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:20.319108    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9rj8d_941899a6-9517-410e-b439-94c967f6c162/global-pull-secret-syncer/0.log"
Sep 18 21:14:20.531331 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:20.531301    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rlvbw_86c8b3bd-a424-4004-9c8d-36ae6030370b/konnectivity-agent/0.log"
Sep 18 21:14:20.552284 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:20.552266    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-79.ec2.internal_bc3d8b9722788fbfdcb8bc7f7b68e6ec/haproxy/0.log"
Sep 18 21:14:25.082645 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.082617    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c3894f6-14c5-4135-a98b-f5b78c8e809c/alertmanager/0.log"
Sep 18 21:14:25.109618 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.109597    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c3894f6-14c5-4135-a98b-f5b78c8e809c/config-reloader/0.log"
Sep 18 21:14:25.135102 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.135082    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c3894f6-14c5-4135-a98b-f5b78c8e809c/kube-rbac-proxy-web/0.log"
Sep 18 21:14:25.166513 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.166493    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c3894f6-14c5-4135-a98b-f5b78c8e809c/kube-rbac-proxy/0.log"
Sep 18 21:14:25.196344 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.196328    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c3894f6-14c5-4135-a98b-f5b78c8e809c/kube-rbac-proxy-metric/0.log"
Sep 18 21:14:25.224523 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.224503    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c3894f6-14c5-4135-a98b-f5b78c8e809c/prom-label-proxy/0.log"
Sep 18 21:14:25.253766 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.253748    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1c3894f6-14c5-4135-a98b-f5b78c8e809c/init-config-reloader/0.log"
Sep 18 21:14:25.337589 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.337531    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-6cdhw_9531a7cb-4020-4d03-bde2-df7edfb474ec/kube-state-metrics/0.log"
Sep 18 21:14:25.363739 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.363715    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-6cdhw_9531a7cb-4020-4d03-bde2-df7edfb474ec/kube-rbac-proxy-main/0.log"
Sep 18 21:14:25.395981 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.395960    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-6cdhw_9531a7cb-4020-4d03-bde2-df7edfb474ec/kube-rbac-proxy-self/0.log"
Sep 18 21:14:25.434653 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.434636    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-847f7f5f44-wqqzc_a82816f1-006f-4fe6-bc8b-e63beb86b5b5/metrics-server/0.log"
Sep 18 21:14:25.467987 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.467968    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-q5w68_f6f1e772-aee4-4faa-bf29-c5bd39845eb4/monitoring-plugin/0.log"
Sep 18 21:14:25.593325 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.593263    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gp7m9_d7ae1a48-dec2-4ae8-9e44-02402f265518/node-exporter/0.log"
Sep 18 21:14:25.625907 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.625887    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gp7m9_d7ae1a48-dec2-4ae8-9e44-02402f265518/kube-rbac-proxy/0.log"
Sep 18 21:14:25.648967 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.648947    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gp7m9_d7ae1a48-dec2-4ae8-9e44-02402f265518/init-textfile/0.log"
Sep 18 21:14:25.765744 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.765722    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-pw4h6_00bf8807-5f69-4594-b985-f46b1423b94c/kube-rbac-proxy-main/0.log"
Sep 18 21:14:25.787625 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.787580    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-pw4h6_00bf8807-5f69-4594-b985-f46b1423b94c/kube-rbac-proxy-self/0.log"
Sep 18 21:14:25.808425 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:25.808410    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-pw4h6_00bf8807-5f69-4594-b985-f46b1423b94c/openshift-state-metrics/0.log"
Sep 18 21:14:26.174958 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:26.174930    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ff99f5d4b-rgqq9_f2cd9e5c-88b5-4c16-882d-4fe793b9d02c/telemeter-client/0.log"
Sep 18 21:14:26.197134 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:26.197108    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ff99f5d4b-rgqq9_f2cd9e5c-88b5-4c16-882d-4fe793b9d02c/reload/0.log"
Sep 18 21:14:26.218014 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:26.217986    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6ff99f5d4b-rgqq9_f2cd9e5c-88b5-4c16-882d-4fe793b9d02c/kube-rbac-proxy/0.log"
Sep 18 21:14:26.253621 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:26.253600    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d85db47d-29rrf_f1ed5f4a-9853-4d69-872f-cfb6074afecc/thanos-query/0.log"
Sep 18 21:14:26.279048 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:26.279026    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d85db47d-29rrf_f1ed5f4a-9853-4d69-872f-cfb6074afecc/kube-rbac-proxy-web/0.log"
Sep 18 21:14:26.300143 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:26.300107    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d85db47d-29rrf_f1ed5f4a-9853-4d69-872f-cfb6074afecc/kube-rbac-proxy/0.log"
Sep 18 21:14:26.324518 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:26.324501    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d85db47d-29rrf_f1ed5f4a-9853-4d69-872f-cfb6074afecc/prom-label-proxy/0.log"
Sep 18 21:14:26.347901 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:26.347878    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d85db47d-29rrf_f1ed5f4a-9853-4d69-872f-cfb6074afecc/kube-rbac-proxy-rules/0.log"
Sep 18 21:14:26.381894 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:26.381871    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d85db47d-29rrf_f1ed5f4a-9853-4d69-872f-cfb6074afecc/kube-rbac-proxy-metrics/0.log"
Sep 18 21:14:28.639652 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:28.639626    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d9ddcb57d-x8jtg_241d9e9c-0fc0-4189-a6b7-84cf0be0a364/console/0.log"
Sep 18 21:14:29.428738 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.428704    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"]
Sep 18 21:14:29.429046 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.429035    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e78759-1efd-4106-b995-d945e25daa2b" containerName="storage-initializer"
Sep 18 21:14:29.429096 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.429048    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e78759-1efd-4106-b995-d945e25daa2b" containerName="storage-initializer"
Sep 18 21:14:29.429096 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.429066    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e78759-1efd-4106-b995-d945e25daa2b" containerName="storage-initializer"
Sep 18 21:14:29.429096 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.429072    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e78759-1efd-4106-b995-d945e25daa2b" containerName="storage-initializer"
Sep 18 21:14:29.429257 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.429157    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e78759-1efd-4106-b995-d945e25daa2b" containerName="storage-initializer"
Sep 18 21:14:29.432332 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.432315    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.435070 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.435046    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2nxhr\"/\"kube-root-ca.crt\""
Sep 18 21:14:29.435241 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.435072    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2nxhr\"/\"default-dockercfg-n49wm\""
Sep 18 21:14:29.435241 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.435173    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2nxhr\"/\"openshift-service-ca.crt\""
Sep 18 21:14:29.441029 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.441005    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"]
Sep 18 21:14:29.527820 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.527791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-proc\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.527820 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.527823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c9z6\" (UniqueName: \"kubernetes.io/projected/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-kube-api-access-8c9z6\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.528057 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.527852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-sys\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.528057 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.527989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-lib-modules\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.528186 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.528057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-podres\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.628973 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.628945    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-lib-modules\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.629180 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.628995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-podres\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.629180 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.629146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-lib-modules\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.629180 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.629171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-proc\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.629314 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.629191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9z6\" (UniqueName: \"kubernetes.io/projected/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-kube-api-access-8c9z6\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.629314 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.629190    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-podres\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.629314 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.629215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-sys\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.629314 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.629273    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-proc\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.629314 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.629279    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-sys\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.636862 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.636836    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c9z6\" (UniqueName: \"kubernetes.io/projected/d9af5cc2-a7f5-4890-b25d-7adb2790c91e-kube-api-access-8c9z6\") pod \"perf-node-gather-daemonset-fsv6w\" (UID: \"d9af5cc2-a7f5-4890-b25d-7adb2790c91e\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:29.742174 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:29.742064    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:30.013343 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:30.013222    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"]
Sep 18 21:14:30.015467 ip-10-0-130-79 kubenswrapper[2583]: W0918 21:14:30.015442    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd9af5cc2_a7f5_4890_b25d_7adb2790c91e.slice/crio-18f402084967624309cda2f3e30afb372c9933e702659b8fb16998cbec085c1f WatchSource:0}: Error finding container 18f402084967624309cda2f3e30afb372c9933e702659b8fb16998cbec085c1f: Status 404 returned error can't find the container with id 18f402084967624309cda2f3e30afb372c9933e702659b8fb16998cbec085c1f
Sep 18 21:14:30.037735 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:30.037709    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k6v7r_3e375926-9dc6-49d2-865b-fa57f024b925/dns/0.log"
Sep 18 21:14:30.061523 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:30.061497    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k6v7r_3e375926-9dc6-49d2-865b-fa57f024b925/kube-rbac-proxy/0.log"
Sep 18 21:14:30.208393 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:30.208372    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jt29h_1003b80d-d1eb-4bf7-a93d-96fc66cdd16d/dns-node-resolver/0.log"
Sep 18 21:14:30.720471 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:30.720443    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dmm78_4bff1259-e240-4481-a8ea-a6ddc0e0ee6a/node-ca/0.log"
Sep 18 21:14:30.915535 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:30.915502    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w" event={"ID":"d9af5cc2-a7f5-4890-b25d-7adb2790c91e","Type":"ContainerStarted","Data":"9cb22ae8f4381297d055e4dce0fb3440c82bbcb38586eed30f7b9868840f147b"}
Sep 18 21:14:30.916026 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:30.915555    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w" event={"ID":"d9af5cc2-a7f5-4890-b25d-7adb2790c91e","Type":"ContainerStarted","Data":"18f402084967624309cda2f3e30afb372c9933e702659b8fb16998cbec085c1f"}
Sep 18 21:14:30.916026 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:30.915583    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:30.930637 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:30.930588    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w" podStartSLOduration=1.93057352 podStartE2EDuration="1.93057352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:14:30.929273512 +0000 UTC m=+3221.837175794" watchObservedRunningTime="2026-09-18 21:14:30.93057352 +0000 UTC m=+3221.838475853"
Sep 18 21:14:31.859319 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:31.859295    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-plv2v_9ddba8b7-bff2-4785-9db8-ddd7c810c350/serve-healthcheck-canary/0.log"
Sep 18 21:14:32.202497 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:32.202470    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dfc68_a824cb57-ceee-40c5-927e-d3c6bbc13df4/kube-rbac-proxy/0.log"
Sep 18 21:14:32.222559 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:32.222534    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dfc68_a824cb57-ceee-40c5-927e-d3c6bbc13df4/exporter/0.log"
Sep 18 21:14:32.242662 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:32.242641    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dfc68_a824cb57-ceee-40c5-927e-d3c6bbc13df4/extractor/0.log"
Sep 18 21:14:34.662588 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:34.662561    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7dcf4bff6-q8flf_5436145d-779b-42ce-a406-53842b043fbc/manager/0.log"
Sep 18 21:14:34.693021 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:34.692992    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-bhqdm_dd491e0b-5e5b-4c6d-a042-3d11f1fddf31/manager/0.log"
Sep 18 21:14:35.552940 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:35.552914    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-ts8m6_3ef17b10-51f0-4100-9820-55c34ad598a5/seaweedfs/0.log"
Sep 18 21:14:35.601247 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:35.601217    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-knh92_4be1e359-c781-44e2-afc1-b054332f1663/seaweedfs-tls-serving/0.log"
Sep 18 21:14:36.930510 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:36.930484    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-fsv6w"
Sep 18 21:14:41.256851 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:41.256822    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p55j2_4b5e1706-b06b-4fd9-b342-9af08311c3af/kube-multus-additional-cni-plugins/0.log"
Sep 18 21:14:41.279396 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:41.279370    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p55j2_4b5e1706-b06b-4fd9-b342-9af08311c3af/egress-router-binary-copy/0.log"
Sep 18 21:14:41.302602 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:41.302578    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p55j2_4b5e1706-b06b-4fd9-b342-9af08311c3af/cni-plugins/0.log"
Sep 18 21:14:41.323806 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:41.323782    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p55j2_4b5e1706-b06b-4fd9-b342-9af08311c3af/bond-cni-plugin/0.log"
Sep 18 21:14:41.349300 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:41.349239    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p55j2_4b5e1706-b06b-4fd9-b342-9af08311c3af/routeoverride-cni/0.log"
Sep 18 21:14:41.442637 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:41.442604    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p55j2_4b5e1706-b06b-4fd9-b342-9af08311c3af/whereabouts-cni-bincopy/0.log"
Sep 18 21:14:41.461215 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:41.461183    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p55j2_4b5e1706-b06b-4fd9-b342-9af08311c3af/whereabouts-cni/0.log"
Sep 18 21:14:41.500791 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:41.500758    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-frkc9_cf1f94f2-24f9-4064-a595-ae2716f9a047/kube-multus/0.log"
Sep 18 21:14:41.589271 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:41.589243    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wkf7p_eaf22891-e55a-433a-a689-fb5b3e3575f9/network-metrics-daemon/0.log"
Sep 18 21:14:41.610256 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:41.610195    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wkf7p_eaf22891-e55a-433a-a689-fb5b3e3575f9/kube-rbac-proxy/0.log"
Sep 18 21:14:42.558652 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:42.558628    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-controller/0.log"
Sep 18 21:14:42.576298 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:42.576269    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/1.log"
Sep 18 21:14:42.576422 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:42.576304    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovn-acl-logging/0.log"
Sep 18 21:14:42.593349 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:42.593325    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/kube-rbac-proxy-node/0.log"
Sep 18 21:14:42.616604 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:42.616582    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 21:14:42.637424 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:42.637405    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/northd/0.log"
Sep 18 21:14:42.656923 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:42.656906    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/nbdb/0.log"
Sep 18 21:14:42.677813 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:42.677791    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/sbdb/0.log"
Sep 18 21:14:42.717057 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:42.717028    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4xrg_059bd389-2748-46bb-bb0f-6b57cc6d391d/ovnkube-controller/0.log"
Sep 18 21:14:43.918205 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:43.918180    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qvkkm_1ecf386e-6ab3-4448-9c76-75a96b3cfa75/network-check-target-container/0.log"
Sep 18 21:14:44.786417 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:44.786394    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-98jdh_c011d236-b524-45ce-a873-88ca80a4c3d9/iptables-alerter/0.log"
Sep 18 21:14:45.545141 ip-10-0-130-79 kubenswrapper[2583]: I0918 21:14:45.545094    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-nkg67_9f0229ef-b6b0-44da-beac-0973f4229671/tuned/0.log"